Cluster deployment method, device and storage medium

Abstract

The present application discloses a cluster deployment method, device and storage medium, the cluster deployment method comprising: in response to receiving a task to be deployed, determining a corresponding cluster deployment strategy according to the initial deployment information of the task to be deployed; according to the cluster deployment strategy, comparing the target deployment information parsed from the initial deployment information with the candidate device information of each candidate device in the cluster to obtain an information comparison result; determining the candidate device corresponding to the information comparison result with consistent characterization information as the target device; deploying the task to be deployed to the target device. The above scheme can improve the flexibility of the cluster deployment process.

Description

Technical Field

[0001] The present application relates to the field of server technology, and in particular to a cluster deployment method, device, and storage medium. Background Art

[0002] In the field of server technology, HDFS (Hadoop Distributed File System) is a high-reliability, high-performance, distributed file system in the Hadoop ecosystem, which is used to store and manage massive data. Kubernetes provides a unified container orchestration and management platform, which greatly simplifies the deployment and management process of applications.

[0003] In private big data deployment scenarios, due to the large number of big data components and the uneven physical equipment on site, more and more manufacturers are beginning to deploy big data on Kubernetes to simplify the complexity of on-site implementation personnel in deploying big data clusters.

[0004] However, for stateful services such as HDFS that rely heavily on local storage, implementers usually deploy storage at a scale that can be met by on-site physical devices. This approach will limit the cluster to the worst device, resulting in a waste of disk space, inability to monopolize disks, and difficulty in subsequent expansion. Summary of the invention

[0005] The present application at least provides a cluster deployment method, apparatus, device, and computer-readable storage medium.

[0006] The first aspect of the present application provides a cluster deployment method, including: in response to receiving a task to be deployed, determining a corresponding cluster deployment strategy according to the initial deployment information of the task to be deployed; according to the cluster deployment strategy, comparing the target deployment information parsed from the initial deployment information with the candidate device information of each candidate device in the cluster to obtain an information comparison result; determining the candidate device corresponding to the information comparison result with consistent characterization information as a target device; and deploying the task to be deployed to the target device.

[0007] In one embodiment, in response to receiving the task to be deployed, the initial deployment information includes function annotation information, and the corresponding cluster deployment strategy is determined according to the initial deployment information of the task to be deployed, including: judging whether a preset device delegation function is enabled according to the function annotation information; if enabled, the target device label information and device stain tolerance information of the task to be deployed in the initial deployment information are used as the target deployment information for cluster deployment; if not enabled, the target device attribute information of the task to be deployed in the initial deployment information is used as the target deployment information for cluster deployment.

[0008] In one embodiment, the candidate device information includes candidate device label information and device stain information. After the target device label information and device stain tolerance information of the task to be deployed in the initial deployment information are used as the target deployment information for cluster deployment, the method includes: comparing the target device label information with the candidate device label information to obtain a label comparison result; comparing the device stain tolerance information with the device stain information to obtain a stain tolerance comparison result; in response to the label comparison result indicating that the target device label information is consistent with the candidate device label information, and the stain tolerance comparison result indicating that the device stain tolerance information is consistent with the device stain information, determining that the information is consistent, and obtaining the information comparison result.

[0009] In one embodiment, the candidate device information includes device attribute information. After cluster deployment is performed using the target device attribute information of the task to be deployed in the initial deployment information as the target deployment information, the method includes: comparing the device attribute information with the target device attribute information; in response to the device attribute information and the target device attribute information being consistent, determining that the information is consistent, and obtaining the information comparison result.

[0010] In one embodiment, after deploying the task to be deployed to the target device, the method further includes: determining whether a preset hierarchical storage function is enabled based on the function annotation information; if enabled, adding the storage medium information of the target device to the task to be deployed, so that the task to be deployed performs data storage based on the storage medium information.

[0011] In one embodiment, the storage medium information includes a storage medium type, and adding the storage medium information of the target device to the task to be deployed so that the task to be deployed performs data storage according to the storage medium information includes: adding the storage medium type of the target device to a copy of the task to be deployed so that the task to be deployed selects a target device of a corresponding storage medium type for data storage according to the amount of data to be processed.

[0012] In one embodiment, the number of the target devices is multiple, and deploying the task to be deployed to the target devices includes: obtaining the number of copies of the task to be deployed in the initial deployment information; in response to the number of copies being at least two and detecting the copy anti-affinity configuration information in the initial deployment information, deploying each copy of the task to be deployed to each target device respectively.

[0013] In one embodiment, after deploying the task to be deployed to the target device, the method further includes: determining the task to be deployed after the target device as a deployed service; in response to the detection of a new device, increasing the number of copies of the deployed service according to the number of the new devices to obtain a new copy of the task to be deployed; and deploying the new copy to the new device.

[0014] The second aspect of the present application provides a cluster deployment device, including: a strategy determination module, which is used to determine the corresponding cluster deployment strategy according to the initial deployment information of the task to be deployed in response to receiving the task to be deployed; an information comparison module, which is used to compare the target deployment information parsed from the initial deployment information with the candidate device information of each candidate device in the cluster according to the cluster deployment strategy to obtain an information comparison result; a device determination module, which is used to determine the candidate device corresponding to the information comparison result with consistent characterization information as the target device; and a task deployment module, which is used to deploy the task to be deployed to the target device.

[0015] A third aspect of the present application provides an electronic device, including a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the above-mentioned cluster deployment method.

[0016] A fourth aspect of the present application provides a computer-readable storage medium on which program instructions are stored, and when the program instructions are executed by a processor, the above-mentioned cluster deployment method is implemented.

[0017] The above scheme, when receiving the task to be deployed, determines the corresponding cluster deployment strategy according to the initial deployment information of the task to be deployed; according to the cluster deployment strategy, compares the target deployment information parsed from the initial deployment information with the candidate device information of each candidate device in the cluster to obtain an information comparison result, thereby determining the candidate device that meets the deployment requirements of the task to be deployed, and determines the candidate device corresponding to the information comparison result with consistent characterization information as the target device; deploying the task to be deployed to the target device can avoid being affected by other devices in the cluster when deploying the task to be deployed to the target device, thereby improving the flexibility of cluster deployment and optimizing the cluster deployment effect.

[0018] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0020] Figure 1 It is a flowchart of an exemplary embodiment of the cluster deployment method of the present application;

[0021] Figure 2 It is a simple schematic diagram of service scheduling to nodes in the cluster deployment method of this application;

[0022] Figure 3 is a block diagram of a cluster deployment device shown in an exemplary embodiment of the present application;

[0023] Figure 4 It is a structural schematic diagram of an embodiment of the electronic device of the present application;

[0024] Figure 5 It is a structural diagram of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0025] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0026] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0027] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the objects associated before and after are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0028] For ease of understanding, one of the applicable scenarios of this application is now exemplified. Taking the task to be deployed as a data node (Datanode) service as an example, the Datanode data node is a slave node in the HDFS architecture, which is mainly responsible for storing and managing data. In the traditional cluster deployment method, when the implementer deploys the HDFS component directly in the cloud native environment of Kubernetes, it is necessary to declare the storage media information used by the Datanode service (such as the number of disks, disk media type and disk capacity, etc.), and then determine the number of Datanode services to be deployed based on the number of physical machines.

[0029] In a cloud-native environment, in order to ensure data security and application security, the same service will be deployed in the form of replicas when deployed. That is, the configuration information of each Datanode service after expansion is the same, and each replica of the same service will use the same storage medium information.

[0030] However, in actual application scenarios, the models and specifications of various storage media in the cluster vary, which means that the types, numbers, and even capacities of disks that can be mounted on various replicas of the same service are different. At this time, the implementer can only deploy the Datanode service at a scale where all storage media in the cluster of the current application scenario can be successfully deployed. Limited by the influence of low-performance machines in the current cluster (for example, the capacity of some storage media in the cluster is small or the types of some storage media are different), when deploying services at a lower standard, other high-performance machines will waste disk space, the service will not be able to monopolize the disk, and it will also lead to many problems such as difficulty in subsequent expansion.

[0031] Based on the existing problems, this application proposes a cluster deployment method based on cloud-native big data heterogeneous storage to solve the above problems, allowing the service Pod to be deployed to exclusively occupy the entire disk corresponding to the physical machine, so as to achieve the purpose of mounting the replicas of the same service on different disks, and to achieve different replicas of the Datanode service mounted in the target device with different storage media information, without paying attention to the differences between different physical machines, and making full use of the storage resources of the cluster. Among them, Pod refers to the smallest deployment and management unit in Kubernetes, which is a collection of one or more containers that can share the same network and storage resources.

[0032] See also Figure 1 , Figure 1 This is a flow chart of an exemplary embodiment of the cluster deployment method of the present application. The cluster deployment method of the present application can be applied to the cluster to be deployed, for example, it can be applied to Kubernetes (k8s for short) of the cluster to be deployed. Specifically, it can include the following steps:

[0033] Step S110 , in response to receiving the task to be deployed, determining a corresponding cluster deployment strategy according to the initial deployment information of the task to be deployed.

[0034] The task to be deployed in the present application refers to deploying cluster services in the cluster, which may include but are not limited to data storage services, data management services, etc., such as the Datanode service in the aforementioned example.

[0035] Initial deployment information is information pre-set for the task to be deployed and used to characterize the task deployment requirements. For example, the initial deployment information may include, but is not limited to, the name of the deployed service, the number of service replicas, the CPU size required for the service, the memory size required for the service, the disk capacity and media type required for the service, the anti-affinity configuration of the service replica, and other relevant notes.

[0036] Exemplarily, k8s may obtain initial deployment information from the deployment manifest file in response to the received deployment manifest file. The deployment manifest file may be uploaded to k8s by the implementation personnel, or obtained from other channels, which is not limited in this application. k8s can determine that the service is arranged and scheduled in the form of Pod through the initial deployment information.

[0037] Specifically, there may be one or more types of deployment information in the initial deployment information, and different deployment information may be applicable to different cluster deployment scenarios. Therefore, in this step, it is necessary to determine the current cluster deployment scenario based on the initial deployment information, and thereby determine the corresponding cluster deployment strategy. For example, the initial deployment information includes function annotation information, and the switch status of the relevant functions participating in the cluster deployment in the cluster can be judged based on the function annotation information, thereby selecting the corresponding cluster deployment strategy. Among them, the cluster deployment strategy of the present application may include but is not limited to deployment based on the custom information of the device and / or deployment based on the attribute information of the device, which will not be repeated here.

[0038] Step S120 , according to the cluster deployment strategy, the target deployment information parsed from the initial deployment information is compared with the candidate device information of each candidate device in the cluster to obtain an information comparison result.

[0039] In combination with the above steps, after determining the cluster deployment strategy based on the initial deployment information, the target deployment information in the initial deployment information can be obtained, and the target deployment information can be compared with the candidate device information of each candidate device in the current cluster to obtain an information comparison result, so as to determine the device that supports the deployment service based on the information comparison result.

[0040] Exemplarily, the aforementioned steps have explained that the initial deployment information may include one or more deployment information. If a strategy of deploying according to the attribute information of the device is selected, the device attribute information (such as the number of devices, device media type, and device capacity, etc.) included in the initial deployment information may be used as the target deployment information for information comparison. If a strategy of deploying according to custom information is selected, some custom information (such as custom label identification information, etc.) included in the initial deployment information may be used as the target deployment information for information comparison. Among them, the method of parsing the target deployment information from the initial deployment information may include but is not limited to a method based on string recognition, a method based on information flag bit recognition, etc., which is not limited here.

[0041] Step S130 , determining the candidate device corresponding to the information comparison result with consistent representation information as the target device.

[0042] In conjunction with the above steps, if the candidate device information of one or more candidate devices is consistent with the target deployment information, it indicates that the one or more candidate devices can support the deployment of the task to be deployed. That is, the candidate devices corresponding to the information comparison results with consistent information are determined as target devices that can be deployed.

[0043] It should be noted that the candidate device and target device of the present application can be understood as a storage medium. To facilitate the subsequent understanding and explanation of the present application, the candidate device, target device, etc. used to deploy services in the present application are referred to as disks in the following text, but the device used to deploy services in the present application is not limited to only disks.

[0044] Step S140: deploy the task to be deployed to the target device.

[0045] Exemplarily, the method of deploying the task to be deployed to the target device may refer to the existing cluster service deployment method, such as scheduling the service to be deployed and mounting it to the target device disk, which will not be described in detail here.

[0046] It can be seen that when receiving the task to be deployed, the present application determines the corresponding cluster deployment strategy according to the initial deployment information of the task to be deployed; according to the cluster deployment strategy, the target deployment information parsed from the initial deployment information is compared with the candidate device information of each candidate device in the cluster to obtain an information comparison result, thereby determining the candidate device that meets the deployment requirements of the task to be deployed, and determining the candidate device corresponding to the information comparison result with consistent characterization information as the target device; deploying the task to be deployed to the target device can avoid being affected by other devices in the cluster when deploying the task to be deployed to the target device, thereby improving the flexibility of cluster deployment and optimizing the cluster deployment effect.

[0047] Based on the above embodiment, the embodiment of the present application describes the steps of determining the corresponding cluster deployment strategy according to the initial deployment information of the task to be deployed. The initial deployment information includes function annotation information. Specifically, the method of this embodiment includes the following steps:

[0048] Determine whether the preset device delegation function is enabled based on the function annotation information; if enabled, use the target device label information and device stain tolerance information of the task to be deployed in the initial deployment information as the target deployment information for cluster deployment; if not enabled, use the target device attribute information of the task to be deployed in the initial deployment information as the target deployment information for cluster deployment.

[0049] In conjunction with the foregoing embodiments, the initial deployment information of the present application may include functional annotation information. The functional annotation information may be added in advance in the initial deployment information of the deployment manifest file by the implementation personnel involved in the cluster deployment. Functional annotation information can be used to characterize the switch status of some functions that need to be used in the k8s cluster. For example, functional annotation information may include device (disk) delegation functional annotations (heterogeneous / disks annotations) and hierarchical storage functional annotations (hierarchicalStorage / volumeMedia annotations) or other more functional annotations, which will not be repeated here.

[0050] It should be noted that when adding the disk delegation function annotation, the implementer must also declare the target device label information label and device taint tolerance information tolerance of the device to be deployed. Among them, the device taint tolerance information tolerance usually exists in pairs with the device taint information taints, and is an important mechanism for controlling Pod scheduling. For example, the k8s scheduler can schedule the Pod to a disk device that can tolerate the taint or a disk device without taints. The specific details are not repeated. The implementer can pre-set corresponding labels and taints for the disks in different slots according to the differences between the physical machines in the current cluster to be deployed. If the disks in slots 1, 2, and 3 of node A and the disks in slots 1 and 2 of node B need to be mounted as Datanode data disks when using the cluster deployment method of the present application (for example, when deploying Datanode services), they can be pre-marked with the target device label information and device taint information corresponding to the Datanode service. The labels and taint information of each physical machine disk can be stored in their respective disks.json configuration files.

[0051] Therefore, the disk is labeled and tainted at the same time, which can prevent other services from using the disk when applying for local volumes, thereby achieving the purpose of Datanode service monopolizing the entire disk.

[0052] Exemplarily, after k8s receives the deployment manifest file, it can search for function annotation information from the initial deployment information of the deployment manifest file. For example, it can determine whether the disk delegation function is enabled in the current k8s cluster based on the identified disk delegation function annotation. If the disk delegation function is enabled, the representation can be used for subsequent cluster deployment process based on the target device label information and device stain tolerance information parsed from the initial deployment information as target deployment information.

[0053] As another example, if it is determined based on the identified disk delegation function annotation that the disk delegation function is not enabled in the current k8s cluster, it is necessary to obtain the device attribute information of the target device declared in advance by the implementer in the deployment manifest file (such as the number of disks in the current cluster, disk media type and disk capacity, etc.) as the target deployment information for the subsequent cluster deployment process.

[0054] Based on the above embodiment, the embodiment of the present application describes the steps after cluster deployment using the target device label information and device stain tolerance information of the task to be deployed in the initial deployment information as the target deployment information. Among them, the candidate device information includes the candidate device label information and the device stain information. Specifically, the method of this embodiment includes the following steps:

[0055] The target device label information is compared with the candidate device label information to obtain a label comparison result; the device stain tolerance information is compared with the device stain information to obtain a stain tolerance comparison result; in response to the label comparison result indicating that the target device label information is consistent with the candidate device label information, and the stain tolerance comparison result indicating that the device stain tolerance information is consistent with the device stain information, it is determined that the information is consistent and an information comparison result is obtained.

[0056] In conjunction with the above-mentioned embodiment, if it is determined that the disk delegation function is enabled, the cluster deployment strategy of the custom information can be selected for subsequent cluster deployment. For example, after obtaining the target device label information and device stain tolerance information of the task to be deployed, the target device label information and device stain tolerance information are used as target deployment information to compare with the candidate device information of each candidate device in the cluster to obtain the information comparison result.

[0057] The candidate device information may include, but is not limited to, candidate device label information and device stain information. For details, please refer to the description of the aforementioned embodiment, which will not be described in detail here.

[0058] In this embodiment, the method for comparing the target deployment information with the candidate device information of each candidate device in the cluster includes: comparing the target device label information with the label information of each candidate device to obtain the label comparison result between the target device label of the service to be deployed and the candidate device label of each candidate device; in addition, similarly comparing the device stain tolerance information with the device stain information to obtain the stain tolerance comparison result. It should be noted that the label information comparison process and the stain information comparison process of the present application can be executed simultaneously or successively, which is not limited here.

[0059] In response to the label comparison result indicating that the target device label information is consistent with the candidate device label information, and the stain tolerance comparison result indicating that the device stain tolerance information is consistent with the device stain information, the information is determined to be consistent, and an information comparison result is obtained. The candidate device whose target device label information is consistent with the candidate device label information and whose device stain tolerance information is consistent with the device stain information is determined as the target device for the service to be deployed.

[0060] It should also be noted that if there is no consistent label information after comparing the target device label information with the label information of each candidate device; and / or if there is no consistent stain tolerance information and stain information after comparing the device stain tolerance information with the stain information of each device, it can indicate that there are problems such as deployment information setting errors or disk device installation errors. Therefore, you can choose to generate error information and report it, and / or change the cluster deployment strategy. For example, perform cluster deployment according to the method of comparing device attribute information in the above embodiment.

[0061] Based on the above embodiment, the embodiment of the present application describes the steps of cluster deployment by using the target device attribute information of the task to be deployed in the initial deployment information as the target deployment information. Among them, the candidate device information includes device attribute information. Specifically, the method of this embodiment includes the following steps:

[0062] The device attribute information is compared with the target device attribute information; in response to the device attribute information and the target device attribute information being consistent, it is determined that the information is consistent, and an information comparison result is obtained.

[0063] In conjunction with the above-mentioned embodiment, if the device delegation function is not enabled, the device attribute information can be selected as the reference information for the cluster deployment strategy. Among them, the candidate device information may also include device attribute information. The device attribute information of this application refers to the attribute-related information that can be obtained from the cluster device (disk), such as the number of disks in the cluster, the storage medium type of each disk, and the disk capacity of each disk, etc. Accordingly, these device attribute information will also be added in the deployment manifest file.

[0064] Similar to the above-mentioned embodiment, in this embodiment, the device attribute information is compared with the target device attribute information; in response to the device attribute information and the target device attribute information being consistent, the information is determined to be consistent, and an information comparison result is obtained. Therefore, the candidate device whose device attribute information is consistent with the target device attribute information among the candidate devices can be used as the target device.

[0065] In summary, implementers do not actually need to understand the storage differences between devices in the cluster. At least they can provide one deployment manifest file to mount different disks for different copies of the Datanode service.

[0066] Based on the above embodiments, the present embodiment of the present application describes the steps after the task to be deployed is deployed to the target device. Specifically, the method of this embodiment includes the following steps:

[0067] It is determined whether the preset hierarchical storage function is enabled according to the function annotation information; if enabled, the storage medium information of the target device is added to the task to be deployed, so that the task to be deployed stores data according to the storage medium information.

[0068] In conjunction with the above-mentioned embodiments, the functional annotation information of the present application may also include a hierarchical storage function annotation. The hierarchical storage function refers to the ability to store data in devices of different storage media types in a cluster.

[0069] The storage medium information may include information such as the storage medium type and mounting path of the storage medium.

[0070] Exemplarily, the storage medium type of the target device of the present application may include but is not limited to a mechanical hard disk (HDD) and / or a solid state drive (SSD). If k8s determines that the hierarchical storage function is enabled based on the function annotation information, the disk medium type and disk mount path of each slot can be automatically injected into the task to be deployed in the form of environment variables (for example, injected into the replica Pod of the Datanode service that is pulled up).

[0071] It is understandable that the cluster deployment method of this application may change the configuration information in the original cluster. Therefore, the service deployed in the cluster can obtain the persistent directory and corresponding storage medium mounted in the Pod by reading the value of the environment variable, and replace it with the configuration file of the service after being sorted by the system, so as to realize the reconfiguration of the cluster, thus completing the hierarchical storage function of docking with the open source HFDS.

[0072] After the storage media information of each disk in the current cluster is injected into the replica Pod of the Datanode, the Pod can select the appropriate storage medium to perform data storage processing on the received data to be stored according to the storage media information of the mounted disk when running.

[0073] Based on the above embodiment, the embodiment of the present application describes the steps of adding the storage medium information of the target device to the task to be deployed so that the task to be deployed stores data according to the storage medium information. The storage medium information includes the storage medium type. Specifically, the method of this embodiment includes the following steps:

[0074] The storage medium type of the target device is added to the copy of the task to be deployed, so that the task to be deployed selects the target device of the corresponding storage medium type for data storage according to the amount of data to be processed.

[0075] Therefore, when the Datanode service obtains the data to be stored, it can select the corresponding disk medium type according to the amount of data to be stored, and then store the data to be stored in the disk of the corresponding disk medium type. For example, it can be judged by setting one or more data volume thresholds. If the amount of data to be stored is less than or equal to a certain data volume threshold, the Datanode service in the cluster can store it in a solid-state hard disk for data storage; if the amount of data to be stored is greater than a certain data volume threshold, the Datanode service in the cluster can store it in a mechanical hard disk for data storage. In this way, the mechanical hard disk in the cluster is used to store large-volume data, and the solid-state hard disk in the cluster is used to store small-volume data.

[0076] Optionally, in addition to enabling the Datanode service to select a corresponding storage medium type for data storage according to the data volume, the above scheme may also select a corresponding storage medium type for data storage according to data access frequency and access frequency threshold (for example, data with low access frequency is stored in a mechanical hard disk, and data with high access frequency is stored in a solid-state drive), and / or select a corresponding storage medium type for data storage according to the data security level (for example, data with a high data security level can be stored in a mechanical hard disk, and data with a low data security level can be stored in a solid-state drive), and / or select a corresponding storage medium type for data storage according to other custom storage types, which are not limited here.

[0077] Based on the above embodiment, the embodiment of the present application describes the steps of deploying the task to be deployed to the target device. There are multiple target devices. Specifically, the method of this embodiment includes the following steps:

[0078] The number of copies of the task to be deployed in the initial deployment information is obtained; in response to the number of copies being at least two and the copy anti-affinity configuration information in the initial deployment information being detected, each copy of the task to be deployed is deployed to each target device respectively.

[0079] In conjunction with the above-mentioned embodiment, in order to ensure that different service replicas can be scheduled to different nodes respectively, it is also necessary to set anti-affinity between different replicas.

[0080] Exemplarily, first, the number of replicas of the task to be deployed in the initial deployment information needs to be obtained, such as the number of replicas of the Datanode service. It is understandable that if the number of replicas is 0 or 1, there is no need to consider the problem of configuring multiple replicas separately. In response to the number of replicas being at least two and the replica anti-affinity configuration information in the initial deployment information being detected, each replica of the task to be deployed is deployed to each target device respectively.

[0081] For example, combined with Figure 2 To explain, Figure 2 It is a simple schematic diagram of service scheduling to nodes in the cluster deployment method of this application. Figure 2 The deployment of Datanode service (DN service for short) and Kafka service in the cluster is shown in Figure 2. Taking Datanode service as an example, if the number of copies of Datanode service is 2, then based on the example of the aforementioned embodiment, k8s will schedule two copies of Datanode service to node A and node B respectively, and declare and mount the disks in slots 1, 2, and 3 for the Datanode service on node A according to the disk label and tolerable stain information, and declare and mount the disks in slots 1 and 2 for the Datanode service on node B. Therefore, the implementation personnel do not need to perceive the differences between the storage media in the cluster, and at least can use one deployment manifest file to achieve different disks mounted for different copies of Datanode service.

[0082] Based on the above embodiments, the present embodiment of the present application describes the steps after the task to be deployed is deployed to the target device. Specifically, the method of this embodiment includes the following steps:

[0083] Determine the tasks to be deployed after the target device as deployed services; in response to the detected new devices, increase the number of copies of the deployed services according to the number of the new devices to obtain new copies of the tasks to be deployed; and deploy the new copies to the new devices.

[0084] In conjunction with the above embodiments, after the tasks to be deployed are deployed to the target device through the methods provided in the above embodiments, the preliminary cluster configuration is completed. However, in fact, the storage media in the cluster will also have certain changes to meet the expansion and contraction requirements of the cluster.

[0085] For ease of explanation, the to-be-deployed tasks deployed after the target device in the aforementioned embodiment are determined as deployed services. When the cluster is expanded, if new devices are detected in the cluster, the number of replicas of the Datanode service can be increased according to the number of new devices to obtain new replicas. The new replicas are then deployed to the new devices accordingly.

[0086] Through the above method, in the specific implementation process, the implementation personnel can add the disk planning file (such as the label information, stain information, attribute information, etc. in the above embodiment) to the cluster physical machine to be expanded as needed, and then incorporate the physical machine into the current Kubernetes cluster. After that, the cluster system will increase the number of replicas of the Datanode service to the number of currently expanded physical machines, and the newly added Datanode service Pod will directly mount the planned disk. Therefore, the implementation personnel only need to pay attention to the number of increased physical machines and disk planning, and do not need to pay attention to the disk differences between the physical machines in the cluster.

[0087] In summary, the cluster deployment method of the present application sinks the local disk scheduling capability to Kubernetes, and on the basis of maintaining the original deployment method, expands a cluster deployment mode that allows cluster services to exclusively occupy local disks, thereby realizing the allocation of local storage resources. By parsing the functional annotation information carried in the deployment manifest file, Kubernetes ignores the volume field, and the disk label on the physical machine managed by Kubernetes determines the number of disks mounted on each Datanode, the disk size, and the disk medium. Through the replica anti-affinity configuration information carried in the deployment manifest file, the Datanode replicas can be deployed on different physical machines, and the Datanode service can be deployed using a deployment manifest file, which automatically adapts to different disk conditions on the physical machine, so that the implementation personnel do not need manual intervention and fully utilize all the disk resources of the physical machine.

[0088] Thanks to the disk delegation and hierarchical storage capabilities in the aforementioned embodiments, implementers can use one deployment manifest file to mount different disk sizes, numbers of disks, and disk media between different replicas of the Datanode service without having to pay attention to the differences between different physical machines. This makes full use of the storage resources of the cluster and solves the problems of inflexible disk configuration and waste of physical machine storage resources encountered in deploying HDFS clusters in a cloud-native manner.

[0089] It should be further explained that the execution subject of the cluster deployment method may be a cluster deployment device, for example, the cluster deployment method may be executed by a terminal device or a server or other processing device, wherein the terminal device may be a user equipment (UE), a computer, a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. In some possible implementations, the cluster deployment method may be implemented by a processor calling a computer-readable instruction stored in a memory.

[0090] Figure 3 FIG. 1 is a block diagram of a cluster deployment device shown in an exemplary embodiment of the present application. Figure 3 As shown, the exemplary cluster deployment device 300 includes: a strategy determination module 310, an information comparison module 320, a device determination module 330 and a task deployment module 340. Specifically:

[0091] The strategy determination module 310 is used to determine the corresponding cluster deployment strategy according to the initial deployment information of the task to be deployed in response to receiving the task to be deployed.

[0092] The information comparison module 320 is used to compare the target deployment information parsed from the initial deployment information with the candidate device information of each candidate device in the cluster according to the cluster deployment strategy to obtain an information comparison result.

[0093] The device determination module 330 is used to determine the candidate device corresponding to the information comparison result with consistent representation information as the target device.

[0094] The task deployment module 340 is used to deploy the task to be deployed to the target device.

[0095] In this exemplary cluster deployment device, when a task to be deployed is received, a corresponding cluster deployment strategy is determined according to the initial deployment information of the task to be deployed; according to the cluster deployment strategy, the target deployment information parsed from the initial deployment information is compared with the candidate device information of each candidate device in the cluster to obtain an information comparison result, thereby determining the candidate device that meets the deployment requirements of the task to be deployed, and determining the candidate device corresponding to the information comparison result with consistent characterization information as the target device; deploying the task to be deployed to the target device can avoid being affected by other devices in the cluster when deploying the task to be deployed to the target device, thereby improving the flexibility of cluster deployment and optimizing the cluster deployment effect.

[0096] It should be noted that the device provided in the above embodiment and the method provided in the above embodiment belong to the same concept, wherein the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment, and will not be repeated here. In practical applications, the device provided in the above embodiment can distribute the above functions to different functional modules as needed, that is, divide the internal structure of the device into different functional modules to complete all or part of the functions described above, and this is not limited here.

[0097] The functions of each module can be found in the cluster deployment method implementation example, which will not be repeated here.

[0098] See also Figure 4 , Figure 4 1 is a schematic diagram of the structure of an embodiment of an electronic device of the present application. The electronic device 100 includes a memory 101 and a processor 102, and the processor 102 is used to execute program instructions stored in the memory 101 to implement the steps in any of the above cluster deployment method embodiments. In a specific implementation scenario, the electronic device 100 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 100 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0099] Specifically, the processor 102 is used to control itself and the memory 101 to implement the steps in any of the above-mentioned cluster deployment method embodiments. The processor 102 can also be called a CPU (Central Processing Unit). The processor 102 may be an integrated circuit chip with signal processing capabilities. The processor 102 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 102 can be implemented by an integrated circuit chip.

[0100] In this exemplary electronic device, when a task to be deployed is received, a corresponding cluster deployment strategy is determined based on the initial deployment information of the task to be deployed; based on the cluster deployment strategy, the target deployment information parsed from the initial deployment information is compared with the candidate device information of each candidate device in the cluster to obtain an information comparison result, thereby determining the candidate device that meets the deployment requirements of the task to be deployed, and determining the candidate device corresponding to the information comparison result with consistent characterization information as the target device; deploying the task to be deployed to the target device can avoid being affected by other devices in the cluster when deploying the task to be deployed to the target device, thereby improving the flexibility of cluster deployment and optimizing the cluster deployment effect.

[0101] See also Figure 5 , Figure 5 The computer-readable storage medium 110 stores program instructions 111 that can be executed by a processor, and the program instructions 111 are used to implement the steps in any of the above cluster deployment method embodiments.

[0102] In this exemplary storage medium, by running the program instructions in the storage medium, when receiving the task to be deployed, the corresponding cluster deployment strategy is determined according to the initial deployment information of the task to be deployed; according to the cluster deployment strategy, the target deployment information parsed from the initial deployment information is compared with the candidate device information of each candidate device in the cluster to obtain an information comparison result, thereby determining the candidate device that meets the deployment requirements of the task to be deployed, and determining the candidate device corresponding to the information comparison result with consistent characterization information as the target device; deploying the task to be deployed to the target device can avoid being affected by other devices in the cluster when deploying the task to be deployed to the target device, thereby improving the flexibility of cluster deployment and optimizing the cluster deployment effect.

[0103] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0104] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0105] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0106] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of a software functional unit. If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A cluster deployment method, characterized in that: The method comprises: In response to receiving the task to be deployed, determining a corresponding cluster deployment strategy according to initial deployment information of the task to be deployed; the initial deployment information includes function annotation information; The step of determining the corresponding cluster deployment strategy according to the initial deployment information of the task to be deployed includes: judging whether the preset device delegation function is enabled according to the function annotation information; if enabled, using the target device label information and device stain tolerance information of the task to be deployed in the initial deployment information as the target deployment information for cluster deployment; if not enabled, using the target device attribute information of the task to be deployed in the initial deployment information as the target deployment information for cluster deployment; According to the cluster deployment strategy, the target deployment information parsed from the initial deployment information is compared with the candidate device information of each candidate device in the cluster to obtain an information comparison result; Determine the candidate device corresponding to the information comparison result with consistent representation information as the target device; Deploy the task to be deployed to the target device.

2. The method according to claim 1, characterized in that The candidate device information includes candidate device label information and device stain information. After cluster deployment is performed using the target device label information and device stain tolerance information of the task to be deployed in the initial deployment information as the target deployment information, the method includes: Compare the target device label information with the candidate device label information to obtain a label comparison result; Comparing the device stain tolerance information with the device stain information to obtain a stain tolerance comparison result; In response to the label comparison result indicating that the target device label information is consistent with the candidate device label information, and the stain tolerance comparison result indicating that the device stain tolerance information is consistent with the device stain information, it is determined that the information is consistent, and the information comparison result is obtained.

3. The method according to claim 1, characterized in that The candidate device information includes device attribute information. After cluster deployment is performed using the target device attribute information of the task to be deployed in the initial deployment information as the target deployment information, the method includes: Comparing the device attribute information with the target device attribute information; In response to the device attribute information being consistent with the target device attribute information, it is determined that the information is consistent, and the information comparison result is obtained.

4. The method according to claim 1, characterized in that After deploying the task to be deployed to the target device, the method further includes: Determining whether a preset hierarchical storage function is enabled according to the function annotation information; If it is turned on, the storage medium information of the target device is added to the task to be deployed, so that the task to be deployed stores data according to the storage medium information.

5. The method according to claim 4, characterized in that The storage medium information includes a storage medium type, and the adding the storage medium information of the target device to the task to be deployed so that the task to be deployed stores data according to the storage medium information includes: The storage medium type of the target device is added to the copy of the task to be deployed, so that the task to be deployed selects a target device of a corresponding storage medium type for data storage according to the amount of data to be processed.

6. The method according to claim 1, characterized in that There are multiple target devices, and deploying the to-be-deployed task to the target devices includes: Obtaining the number of copies of the task to be deployed in the initial deployment information; In response to the number of replicas being at least two and replica anti-affinity configuration information in the initial deployment information being detected, each replica of the task to be deployed is deployed to each target device respectively.

7. The method according to claim 1, characterized in that After deploying the task to be deployed to the target device, the method further includes: Determine the to-be-deployed task deployed after the target device as a deployed service; In response to the detected newly added devices, increasing the number of replicas of the deployed service according to the number of the newly added devices, to obtain newly added replicas of the task to be deployed; Deploy the newly added replica to the newly added device.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the processor is used to execute program instructions stored in the memory to implement the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images