Cluster deployment method and device, electronic equipment and computer readable storage medium

Abstract

The invention provides a cluster deployment method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: carrying out the topology analysis through employing a first neural network according to the intention of a user, and carrying out the capacity prediction through employing a second neural network; inputting the topological structure output by the first neural network and the capacity demand output by the second neural network into an affinity engine to generate target configuration; and deploying the target cluster according to the target configuration. In the embodiment of the invention, topology analysis and capacity prediction are respectively carried out on the user intention through the first neural network and the second neural network, and then the affinity scoring is carried out through the affinity engine to determine the target configuration of the target cluster, so that the automatic configuration of the target cluster is realized, the artificial experience is reduced, the configuration efficiency is improved, and the error rate is reduced.

Description

Technical Field

[0001] The present application relates to the field of cluster configuration, and more specifically, to a cluster deployment method, device, electronic device, and computer-readable storage medium. Background Art

[0002] With the rapid development of cloud computing and distributed systems, cluster deployment has become a core component in supporting large-scale computing power needs. However, current mainstream cluster deployment methods still rely heavily on manual experience, resulting in significant efficiency and reliability issues. In traditional deployment processes, operations and maintenance personnel must manually complete key steps such as resource configuration, component parameter tuning, and network topology design. This is not only time-consuming and tedious, but also prone to configuration errors and resource imbalances due to human negligence. Summary of the Invention

[0003] In view of this, an object of the embodiments of the present application is to provide a cluster deployment method, apparatus, electronic device, and computer-readable storage medium, which can improve configuration efficiency and reduce error rates.

[0004] In a first aspect, an embodiment of the present application provides a cluster deployment method, comprising: performing topology analysis using a first neural network and performing capacity prediction using a second neural network based on user intent; inputting the topology structure output by the first neural network and the capacity requirement output by the second neural network into an affinity engine to generate a target configuration; and deploying a target cluster according to the target configuration.

[0005] In the above implementation process, the first neural network and the second neural network are used to perform topological analysis and capacity prediction on user intentions respectively, and then the affinity engine is used to perform affinity scoring to determine the target configuration of the target cluster, thereby realizing automatic configuration of the target cluster, reducing the need for manual experience, improving configuration efficiency, and reducing error rate.

[0006] In one embodiment, the use of the first neural network for topological analysis includes: obtaining hardware communication indicators; converting the hardware communication indicators into a heterogeneous graph data structure; inputting the heterogeneous graph data structure into a multi-head graph attention network, and calculating the communication path weight through the multi-head graph attention network; and determining the topological classification result based on the communication path weight.

[0007] In this implementation, by converting hardware topology features into a heterogeneous graph data structure, different roles can be directly distinguished through node type labels, and the nature of relationships can be clearly distinguished through edge type labels. This allows for precise expression of the hardware system's multi-type entities and relationships, avoiding information loss. This allows for a more comprehensive and accurate description of the hardware system's complex relationships, enhancing the model's expressiveness and improving the accuracy of analysis. Furthermore, by calculating communication path weights through a heterogeneous graph data structure and a multi-head graph attention network, the communication path weights can be updated in real time based on hardware communication metrics, ensuring that they always match the actual hardware system and improving the accuracy of communication path weight calculations.

[0008] In one embodiment, the use of the second neural network to perform capacity prediction includes: inputting historical hardware data; using a long short-term memory network to forward propagate the historical hardware data and extract the output of the last time step; and predicting the capacity demand within a set time based on the output of the last time step.

[0009] In this implementation, by using a long short-term memory network to propagate historical hardware data forward and predicting capacity requirements within a set timeframe based on the output of the final time step, this effectively captures long-term dependencies in the sequence and improves forecast accuracy. Furthermore, by simultaneously considering data from multiple hardware sources, a more comprehensive understanding of resource usage is achieved, capturing the interactions between different resources and thus improving forecast reliability.

[0010] In one embodiment, after predicting the capacity demand within a set time based on the output of the last time step, the capacity prediction using the second neural network also includes: subjecting the capacity demand to non-negative constraint processing, resetting the elements in the capacity demand whose values ​​are less than a preset threshold to a set value; amplifying the parameters of the specified dimension of the capacity demand by a preset scaling factor; and determining that the capacity demand after the non-negative constraint processing and amplification processing is the actual capacity demand.

[0011] In the above implementation, by applying non-negative constraints to capacity requirements, we can eliminate meaningless negative values ​​that may appear in model predictions, thereby improving the accuracy of capacity requirements. Furthermore, by scaling the parameters of the specified dimensions of the capacity requirements using a preset scaling factor, we can reserve additional space during actual resource allocation, avoiding resource shortages caused by prediction errors, improving the accuracy of capacity requirements, and ultimately increasing resource availability.

[0012] In one embodiment, the topology structure output by the first neural network and the capacity requirement output by the second neural network are input into an affinity engine to generate a target configuration, including: initializing the topology structure and the capacity requirement; converting the topology structure and the capacity requirement into affinity rules for the target cluster; and generating a target configuration according to the affinity rules.

[0013] In the above implementation process, affinity rules that meet user intentions can be automatically generated based on topology results and capacity requirements, which can improve the accuracy of target configuration and thus improve the accuracy of target cluster configuration.

[0014] In one embodiment, before generating the target configuration according to the affinity rules, the method further includes: mapping each rule in the affinity rules to a node of a conflict graph, and establishing node connection edges based on the conflict relationship between the rules to construct the conflict graph; traversing the conflict graph to determine whether there are conflicting rule pairs in the affinity rules; generating the target configuration according to the affinity rules includes: generating the target configuration according to the affinity rules when there are no conflicting rule pairs in the affinity rules.

[0015] In the above implementation process, before generating the target configuration, the affinity rules are first verified to avoid conflicting rules in the affinity rules, thereby avoiding resource waste or scheduling failure and improving the accuracy of the target configuration.

[0016] In one embodiment, deploying the target cluster according to the target configuration includes: generating the topology configuration of the target cluster through an artificial intelligence algorithm based on the target configuration; parsing the topology configuration through a dynamic template engine to generate a mapping relationship between node type and quantity; dynamically generating a corresponding deployment strategy based on the mapping relationship; and deploying the target cluster according to the deployment strategy.

[0017] In this implementation, the optimal topology configuration is automatically generated through an artificial intelligence algorithm, meeting user needs while avoiding resource waste and improving configuration accuracy. Furthermore, a dynamic template engine converts the topology configuration into programmable node mapping rules, enabling dynamic expansion of storage capacity and increasing configuration flexibility. Furthermore, the target cluster is fully automated through scripted deployment and maintenance, enabling automated deployment from resource configuration to software installation, significantly reducing target cluster deployment time and improving deployment efficiency.

[0018] In a second aspect, an embodiment of the present application also provides a cluster deployment device, including: an analysis module for performing topology analysis using a first neural network and capacity prediction using a second neural network based on user intention; a generation module for inputting the topology structure output by the first neural network and the capacity requirement output by the second neural network into an affinity engine to generate a target configuration; and a deployment module for deploying a target cluster according to the target configuration.

[0019] In a third aspect, an embodiment of the present application further provides an electronic device comprising: a processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of the method in the above-mentioned first aspect, or any possible implementation of the first aspect.

[0020] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the cluster deployment method in the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0021] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the following embodiments are given in conjunction with the accompanying drawings for detailed description as follows. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 A block diagram of an electronic device provided in an embodiment of the present application; Figure 2 A flowchart of the cluster deployment method provided in an embodiment of the present application; Figure 3 A specific flow chart of executing the cluster deployment method provided in an embodiment of the present application in a system architecture in which the cluster deployment method is located; Figure 4 A schematic diagram of the functional modules of the cluster deployment device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0024] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application.

[0025] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0026] In the current field of cloud computing and distributed systems, cluster deployment, as a core component supporting large-scale applications, still has significant automation limitations. Traditional deployment models rely primarily on the manual experience of operations and maintenance personnel to complete the following cluster deployments.

[0027] However, after long-term research, the inventors of this application have found that the current cluster deployment methods generally have the following defects: 1. Node planning relies on manual experience and cannot dynamically respond to hardware characteristics; 2. Static capacity planning leads to low resource utilization; 3. Affinity rules need to be written manually, and cross-region deployment configuration has a high error rate.

[0028] In view of this, the present application proposes a cluster deployment method, which performs topological analysis and capacity prediction on user intentions through a first neural network and a second neural network respectively, and then determines the target configuration of the target cluster through affinity scoring through an affinity engine, thereby realizing automatic configuration of the target cluster, reducing the need for manual experience, improving configuration efficiency, and reducing error rate.

[0029] To facilitate understanding of this embodiment, the electronic device that executes the cluster deployment method disclosed in the embodiment of the present application is first introduced in detail.

[0030] like Figure 1 , which is a block diagram of an electronic device. The electronic device 100 may include a memory 111 and a processor 113. A person skilled in the art will understand that Figure 1 The structure shown is only for illustration and does not limit the structure of the electronic device 100. For example, the electronic device 100 may further include Figure 1 More or fewer components than shown, or with Figure 1 Different configurations shown.

[0031] The memory 111 and processor 113 are electrically connected to each other, directly or indirectly, to enable data transmission or interaction. For example, these components may be electrically connected to each other via one or more communication buses or signal lines. The processor 113 is used to execute the executable modules stored in the memory.

[0032] The memory 111 may be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), etc. The memory 111 is used to store programs, and the processor 113 executes the programs after receiving an execution instruction. The method executed by the electronic device 100 defined by the process disclosed in any embodiment of the present application may be applied to the processor 113 or implemented by the processor 113.

[0033] The processor 113 may be an integrated circuit chip with signal processing capabilities. The processor 113 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The methods, steps, and logic block diagrams disclosed in the embodiments of the present application may be implemented or executed. The general-purpose processor may be a microprocessor or any conventional processor.

[0034] The electronic device 100 in this embodiment can be used to execute each step in each method provided in the embodiments of the present application. The implementation process of the cluster deployment method is described in detail below through several embodiments.

[0035] See also Figure 2 , is a flow chart of the cluster deployment method provided by the embodiment of this application. Figure 2 The specific process shown is explained in detail.

[0036] Step 201: Based on user intent, a first neural network is used to perform topology analysis, and a second neural network is used to perform capacity prediction.

[0037] The user intent here can be analyzed using natural language processing technology. That is, after obtaining the user intent, it is analyzed using natural language processing technology, and based on the analysis results, the first neural network is triggered to perform topology analysis, and the second neural network is triggered to perform capacity prediction.

[0038] The first neural network refers to a neural network structure that combines the advantages of topology and neural networks and is capable of processing and analyzing data with complex topological structures. Examples include graph neural networks, topological neural networks, recurrent neural networks, and convolutional neural networks. The first neural network can be selected based on actual circumstances.

[0039] The second neural network refers to a neural network structure that combines time series trend analysis with bottleneck feature identification. For example, a BP neural network, an LSTM neural network, an Elman neural network, etc. The second neural network can be selected based on actual conditions.

[0040] It should be understood that after parsing the user intent, the first neural network can be actively triggered to perform topology analysis based on the user intent parsing result, and the second neural network can be triggered to perform capacity prediction to automatically determine the target configuration of the target cluster.

[0041] Of course, after analyzing the user intent, you can also manually trigger the first neural network to perform topology analysis, and / or manually trigger the second neural network to perform capacity prediction, and then determine the target configuration of the target cluster.

[0042] The topological analysis performed by the first neural network mentioned above can deeply understand the structural characteristics of user intent data, explore the potential relationships between nodes and edges, and provide key information support for related tasks (such as node classification, graph classification, link prediction, etc.).

[0043] The second neural network performs capacity forecasting using a data-driven approach that can accurately estimate the future capacity status of a system or equipment, providing a scientific basis for resource planning, optimal scheduling, and risk management.

[0044] In one embodiment, the first neural network is a graph neural network, and the second neural network is an LSTM neural network.

[0045] Step 202: Input the topology structure output by the first neural network and the capacity requirement output by the second neural network into an affinity engine to generate a target configuration.

[0046] Affinity rules can be set in the affinity engine. The affinity rules can include topological affinity and capacity affinity. Topological affinity can define the affinity of nodes or services based on the topological structure output by the first neural network. Capacity affinity defines the affinity of resource allocation based on the capacity demand predicted by the second neural network.

[0047] In addition, after the configuration plan is determined, each configuration plan can be scored for affinity, and the target configuration of the target cluster can be determined based on the topology and capacity requirements.

[0048] The target cluster here can be a K8s cluster, Docker Swarm, Apache Mesos, Nomad, etc. The target cluster can be selected according to actual conditions.

[0049] It should be understood that after the topological structure is obtained by performing topological analysis through the first neural network, and the capacity demand is output through capacity prediction through the second neural network, the topological structure and capacity demand can be directly input into the affinity engine, and then the affinity engine is used to perform affinity scoring, and then a relatively better configuration scheme is determined, and the relatively better configuration scheme is determined as the target configuration.

[0050] Step 203: deploy the target cluster according to the target configuration.

[0051] In the above implementation process, the first neural network and the second neural network are used to perform topological analysis and capacity prediction on user intentions respectively, and then the affinity engine is used to perform affinity scoring to determine the target configuration of the target cluster, thereby realizing automatic configuration of the target cluster, reducing the need for manual experience, improving configuration efficiency, and reducing error rate.

[0052] In one possible implementation, a first neural network is used to perform topological analysis, including: obtaining hardware communication indicators; converting the hardware communication indicators into a heterogeneous graph data structure; inputting the heterogeneous graph data structure into a multi-head graph attention network, and calculating the communication path weight through the multi-head graph attention network; and determining the topological classification result based on the communication path weight.

[0053] The hardware communication indicators can be obtained from each node. The hardware communication indicators may include features such as GPU model, number of buses and their communication protocols, and PCIE bandwidth. The hardware communication indicators can be selected based on actual conditions.

[0054] Optionally, the hardware communication indicator can be acquired in real time, can be acquired at set time intervals, or can be acquired when triggered by specific conditions. The acquisition method of the hardware communication indicator can be selected according to actual conditions.

[0055] A heterogeneous graph is a graph structure with multiple types of nodes and edges. This heterogeneous graph data structure can be constructed by defining nodes and edges. For example, you can define three types of nodes (GPU, CPU, and storage) and two types of edges (bus and its communication protocol / PCIE).

[0056] Exemplarily, the heterogeneous graph data structure can be constructed in the following ways: defining heterogeneous node types (i.e., determining which types of nodes are needed to represent different entities in the system based on the actual situation of the hardware communication system), defining heterogeneous edge types (i.e., clarifying the association relationships between nodes based on the communication protocol and connection relationship, and using edges to represent these relationships), adding corresponding attributes to each node and edge to store the specific values ​​or levels of hardware communication indicators, and constructing a heterogeneous graph data structure reflecting the hardware communication system based on the above definitions.

[0057] In one embodiment, before converting the hardware communication metrics into a heterogeneous graph data structure, a graph attention network layer of a first neural network may be initialized for processing image data. Additionally, a fully connected layer may be initialized for classification.

[0058] Hardware communication metrics can be used to form a feature matrix. This feature matrix can be processed through a graph attention network layer to generate an embedded representation of the graph. A multi-head attention mechanism aggregates information about neighboring nodes to capture the complex topological relationships in the graph. The classification results are then returned, outputting the probability of each node belonging to each category.

[0059] It can be understood that calculating communication path weights in a multi-head graph attention network can be achieved by initializing multiple attention heads. For each attention head, the attention weights between nodes are independently calculated. Based on the calculated attention weights, the features of neighboring nodes are weighted and aggregated to obtain a new representation for each node. The outputs of each attention head are combined by splicing or averaging to form the final node representation.

[0060] Here, each attention head independently learns the weights of different communication paths.

[0061] In one embodiment, the calculation formula for calculating the communication path weight through the multi-head graph attention network can be as follows: ; in, is the communication path weight, For nodes The eigenvector of is the learnable weight matrix, is the parameter vector of the attention mechanism, is the vector concatenation operation, For nodes The neighbor set of For nodes The eigenvector of For nodes The eigenvector of .

[0062] It should be understood that the embodiments can monitor the hardware communication indicators and promptly update the communication path weights when the hardware communication indicators change. At the same time, the latest hardware communication indicators can also be used to update the parameters of the first neural network.

[0063] The above topological classification results may include dense, dispersed and mixed types, and the topological classification result may be selected according to actual conditions.

[0064] In this implementation, by converting hardware topology features into a heterogeneous graph data structure, different roles can be directly distinguished through node type labels, and the nature of relationships can be clearly distinguished through edge type labels. This allows for precise expression of the hardware system's multi-type entities and relationships, avoiding information loss. This allows for a more comprehensive and accurate description of the hardware system's complex relationships, enhancing the model's expressiveness and improving the accuracy of analysis. Furthermore, by calculating communication path weights through a heterogeneous graph data structure and a multi-head graph attention network, the communication path weights can be updated in real time based on hardware communication metrics, ensuring that they always match the actual hardware system and improving the accuracy of communication path weight calculations.

[0065] In one possible implementation, a second neural network is used to perform capacity prediction, including: inputting historical hardware data; using a long short-term memory network to forward propagate the historical hardware data and extract the output of the last time step; and predicting the capacity demand within a set time based on the output of the last time step.

[0066] The historical hardware data may refer to hardware data within a set time period, for example, hardware data from the past 7 days, hardware data from the past 30 days, hardware data from the past day, etc. The historical hardware data may be selected based on actual conditions.

[0067] Optionally, the historical hardware data may be data such as CPU utilization, memory usage, GPU memory usage, network IO, etc. The historical hardware data may be selected according to actual conditions.

[0068] In one embodiment, the historical hardware data may be input into the second neural network in a specific data format, for example, [batch_size, seq_len=7, features=8].

[0069] It can be understood that the second neural network can perform capacity prediction in the following way: input historical hardware data into the second neural network, learn the long-term dependencies in the time series through a multi-layer long short-term memory network, extract the hidden state of the last time step of each sample, and predict the capacity demand at the next time point through a fully connected layer.

[0070] Forward propagation of historical hardware data using a long short-term memory network can be achieved by inputting the historical hardware data and passing it to the long short-term memory network layer. The long short-term memory network layer captures the long-term dependencies in the time series and outputs the hidden state for each time step. The output of the last time step is then extracted and mapped to multiple target values ​​through a fully connected layer, ultimately outputting multiple predicted values ​​for each sample (for example, three predicted values).

[0071] The above-mentioned setting time can be set according to actual conditions, for example, the next 24 hours, the next 48 hours, the next 36 hours, etc. The setting time can be selected according to actual conditions.

[0072] The capacity requirements here may include memory requirements, CPU requirements, GPU requirements, etc., and the capacity requirements can be selected according to actual conditions.

[0073] In this implementation, by using a long short-term memory network to propagate historical hardware data forward and predicting capacity requirements within a set timeframe based on the output of the final time step, this effectively captures long-term dependencies in the sequence and improves forecast accuracy. Furthermore, by simultaneously considering data from multiple hardware sources, a more comprehensive understanding of resource usage is achieved, capturing the interactions between different resources and thus improving forecast reliability.

[0074] In one possible implementation, after predicting the capacity demand within a set time based on the output of the last time step, using the second neural network to perform capacity prediction also includes: subjecting the capacity demand to non-negative constraints, resetting elements in the capacity demand whose values ​​are less than a preset threshold to a set value; amplifying parameters of a specified dimension of the capacity demand using a preset scaling factor; and determining that the capacity demand after the non-negative constraint processing and amplification processing is the actual capacity demand.

[0075] The preset threshold value here can be set according to the actual situation. For example, 0, 1, 5, 10, etc. The set value can be the same as the preset threshold value or a different value from the preset threshold value. The set value can also be selected according to the actual situation.

[0076] For example, if both the preset threshold and the set value are 0, the non-negativity constraint processing can include replacing values ​​less than 0 in the capacity demand with 0 to ensure the non-negativity of the prediction result. This can eliminate meaningless negative values ​​that may appear in the model prediction. For example, in scenarios where results such as resource demand and price should not be negative.

[0077] The above-mentioned designated dimension refers to a dimension of the capacity requirement, for example, the third column of the capacity requirement, the second column of the capacity requirement, etc. The designated dimension can be specified according to actual needs.

[0078] The preset scaling factor is used to increase the capacity requirement. For example, if the preset scaling factor is 1.2, it provides a 20% buffer for increasing the capacity requirement. If the preset scaling factor is 1.5, it provides a 50% buffer for increasing the capacity requirement, and so on. The preset scaling factor can be selected based on actual conditions.

[0079] It should be understood that by amplifying the parameters of the specified dimension of the capacity demand by a preset scaling factor, additional space can be reserved during actual resource allocation to avoid resource shortages due to prediction errors.

[0080] In one embodiment, after predicting the capacity demand within a set time period based on the output of the last time step, using the second neural network to perform capacity prediction further includes: establishing a nonlinear programming model under constraints to achieve cost-optimal deployment. The nonlinear programming model under constraints can be: ; ; in, for Forecast demand, for Real-time cloud resource unit price, for Real-time cloud resource unit price, for Real-time cloud resource unit price, is the budget constraint, for capacity requirements.

[0081] The cloud resource unit price here can be obtained at a set time interval, or in real time. The method for obtaining the cloud resource unit price can be based on actual conditions.

[0082] It should be understood that if cost is taken into account when determining the target configuration, the following model can be established when predicting capacity requirements: ; in, Reserve weight for CPU, is the memory buffer weight, is the storage IOPS weight, For CPU storage, For memory buffer, For storage.

[0083] Here 、 and It can be adjusted by Bayesian optimization. Used to reflect the priority of CPU-intensive tasks, To cope with sudden memory demands, Used to tune for high-throughput tasks.

[0084] In the above implementation, by applying non-negative constraints to capacity requirements, we can eliminate meaningless negative values ​​that may appear in model predictions, thereby improving the accuracy of capacity requirements. Furthermore, by scaling the parameters of the specified dimensions of the capacity requirements using a preset scaling factor, we can reserve additional space during actual resource allocation, avoiding resource shortages caused by prediction errors, improving the accuracy of capacity requirements, and ultimately increasing resource availability.

[0085] In a possible implementation, step 202 includes: initializing a topology structure and capacity requirements; converting the topology structure and capacity requirements into affinity rules of a target cluster; and generating a target configuration according to the affinity rules.

[0086] Among them, the initialization of the topology structure and capacity requirements may include: performing text classification tasks through a pre-trained Chinese BERT model to convert the topology structure and capacity requirements into target cluster resource constraint specifications.

[0087] For example, load a pre-trained BERT tokenizer and BERT model, use the BERT tokenizer to tokenize the text corresponding to the topology and capacity requirements, and convert the result into a PyTorch tensor. Pass the tokenization result to the BERT model for prediction, and obtain the BERT model output.

[0088] It can be understood that after determining the BERT model output, the BERT model output is passed to the _convert_to_kubernetes_spec method, and the _convert_to_kubernetes_spec method determines which affinity rules to generate based on the value of logits.

[0089] In one embodiment, the affinity rules may be as follows: If logits[0][0]>0.8, cross-zone deployment is required and cross-zone anti-affinity rules should be added.

[0090] If logits[0][1]>0.7, it means that a specific storage type is required and an SSD storage node affinity rule is added.

[0091] The above affinity rules can be directly used to configure the target cluster.

[0092] In the above implementation process, affinity rules that meet user intentions can be automatically generated based on topology results and capacity requirements, which can improve the accuracy of target configuration and thus improve the accuracy of target cluster configuration.

[0093] In one possible implementation, before generating a target configuration based on the affinity rules, the method further includes: mapping each rule in the affinity rules into a node of a conflict graph, and establishing node connection edges based on the conflict relationship between the rules to construct a conflict graph; traversing the conflict graph to determine whether there are conflicting rule pairs in the affinity rules.

[0094] The conflict graph is an undirected graph in which nodes represent rules and edges represent conflicts between rules.

[0095] It should be understood that when creating a conflict graph, you can first create an empty undirected graph and use each rule in the affinity rules as a node in the conflict graph to construct the conflict graph. Then, use the double loop rule to traverse each rule in the affinity rules, checking whether all rule pairs conflict. If a rule pair conflicts, an edge is added to the conflict graph, completing the conflict graph construction.

[0096] After the conflict graph is constructed, it can be further analyzed to determine whether any conflicting affinity rule pairs exist. If so, this indicates that the affinity rules are mutually exclusive, and the affinity can be redefined. If not, this indicates that the affinity rules are not mutually exclusive, and the target configuration can be generated based on these affinity rules.

[0097] The conflicts here can include topology key conflicts and node selector conflicts.

[0098] Topology key conflicts are determined based on the differences in the topologyKeys of two rules (used in a cluster to define the topology field in Pod affinity or anti-affinity rules, thereby controlling Pod scheduling within the cluster). If the topologyKeys of two rules differ (for example, one requires CPU allocation and the other requires memory allocation), a conflict is identified.

[0099] Node selector conflicts can be determined based on the nodeSelector (which specifies the node to which a Pod is scheduled) and selectors of the two rules. If both rules have a nodeSelector and the selectors are incompatible, a conflict occurs.

[0100] In one embodiment, generating the target configuration according to the affinity rules includes: generating the target configuration according to the affinity rules when no conflicting rule pairs exist in the affinity rules.

[0101] In the above implementation process, before generating the target configuration, the affinity rules are first verified to avoid conflicting rules in the affinity rules, thereby avoiding resource waste or scheduling failure and improving the accuracy of the target configuration.

[0102] In one possible implementation, step 203 includes: generating a topology configuration of a target cluster through an artificial intelligence algorithm based on the target configuration; parsing the topology configuration through a dynamic template engine to generate a mapping relationship between node type and quantity; dynamically generating a corresponding deployment strategy based on the mapping relationship; and deploying the target cluster according to the deployment strategy.

[0103] It should be understood that after the target configuration is input into the artificial intelligence algorithm, the artificial intelligence algorithm analyzes the target configuration and determines the topology configuration including information such as cluster topology, resource capacity, affinity rules, etc.

[0104] In one embodiment, the dynamic template engine may be Jinja2.

[0105] The dynamic template engine defines a series of templates for different types of nodes. These templates can contain various node attribute information, such as hardware configuration requirements (such as the number of CPU cores, memory capacity, storage type and capacity), and dependencies between nodes.

[0106] When a topology configuration is entered into the dynamic template engine, it matches the appropriate template based on the node types and quantities described in the topology configuration. Then, through template matching and parsing, the dynamic template engine generates a mapping between node types and quantities. This mapping lists the specific number of nodes required for each type and the corresponding detailed configuration information.

[0107] For ease of understanding, the overall implementation process of the embodiment of the present application is further demonstrated below: Figure 3 As shown, after obtaining the user's intent, the user's intent is parsed using an artificial intelligence algorithm, and the parsing results are transmitted to the topology generation submodule and the capacity planning submodule. The topology analysis is performed by the first neural network in the topology generation submodule, and the capacity prediction is performed by the second neural network in the capacity planning submodule. Then, the topology structure and capacity requirements are input into the affinity engine, which generates the corresponding target configuration. This target configuration is then transmitted to the execution layer, and the execution layer deploys the target cluster according to the target configuration. In addition, the artificial intelligence algorithm can also perform real-time telemetry of the target cluster to promptly detect abnormal conditions in the target cluster.

[0108] In this implementation, the optimal topology configuration is automatically generated through an artificial intelligence algorithm, meeting user needs while avoiding resource waste and improving configuration accuracy. Furthermore, a dynamic template engine converts the topology configuration into programmable node mapping rules, enabling dynamic expansion of storage capacity and increasing configuration flexibility. Furthermore, the target cluster is fully automated through scripted deployment and maintenance, enabling automated deployment from resource configuration to software installation, significantly reducing target cluster deployment time and improving deployment efficiency.

[0109] Based on the same application concept, the embodiment of the present application also provides a cluster deployment device corresponding to the cluster deployment method. Since the principle of solving the problem by the device in the embodiment of the present application is similar to that of the aforementioned cluster deployment method embodiment, the implementation of the device in this embodiment can refer to the description in the embodiment of the above method, and the repeated parts will not be repeated.

[0110] See also Figure 4 , is a functional module diagram of the cluster deployment device provided in the embodiment of the present application. Each module in the cluster deployment device in this embodiment is used to execute each step in the above method embodiment. The cluster deployment device includes an analysis module 301, a generation module 302, and a deployment module 303; wherein, The analysis module 301 is used to perform topology analysis using a first neural network and capacity prediction using a second neural network according to user intent.

[0111] The generation module 302 is used to input the topology structure output by the first neural network and the capacity requirement output by the second neural network into the affinity engine to generate a target configuration.

[0112] The deployment module 303 is used to deploy the target cluster according to the target configuration.

[0113] In one possible implementation, the analysis module 301 is further used to: obtain hardware communication indicators; convert the hardware communication indicators into a heterogeneous graph data structure; input the heterogeneous graph data structure into a multi-head graph attention network, and calculate the communication path weight through the multi-head graph attention network; and determine the topological classification result based on the communication path weight.

[0114] In one possible implementation, the analysis module 301 is further configured to: input historical hardware data; perform forward propagation on the historical hardware data using a long short-term memory network to extract the output of the last time step; and predict capacity requirements within a set time based on the output of the last time step.

[0115] In one possible implementation, the analysis module 301 is specifically used to: subject the capacity demand to non-negative constraint processing, and reset the elements in the capacity demand whose values ​​are less than a preset threshold to a set value; amplify the parameters of the specified dimension of the capacity demand using a preset proportional factor; and determine that the capacity demand after non-negative constraint processing and amplification processing is the actual capacity demand.

[0116] In a possible implementation, the generation module 302 is further configured to: initialize the topology structure and the capacity requirement; convert the topology structure and the capacity requirement into affinity rules of the target cluster; and generate a target configuration according to the affinity rules.

[0117] In one possible implementation, the cluster deployment device also includes a judgment module for mapping each rule in the affinity rules into a node of a conflict graph, and establishing node connection edges based on the conflict relationship between the rules to construct the conflict graph; traversing the conflict graph to determine whether there are conflicting rule pairs in the affinity rules.

[0118] In a possible implementation, the generating module 302 is specifically configured to generate a target configuration according to the affinity rules when no conflicting rule pairs exist in the affinity rules.

[0119] In one possible implementation, the deployment module 303 is also used to: generate the topology configuration of the target cluster through an artificial intelligence algorithm based on the target configuration; parse the topology configuration through a dynamic template engine to generate a mapping relationship between node type and quantity; dynamically generate a corresponding deployment strategy based on the mapping relationship; and deploy the target cluster according to the deployment strategy.

[0120] In addition, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the cluster deployment method described in the above method embodiment are executed.

[0121] The computer program product of the cluster deployment method provided in the embodiment of the present application includes a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the steps of the cluster deployment method described in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0122] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architectures, functions and operations of the devices, methods and computer program products according to the multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, program segment or a part of the code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or action, or can be implemented using a combination of dedicated hardware and computer instructions.

[0123] In addition, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0124] If the functions are implemented in the form of software modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage media include various media that can store program code, such as USB flash drives, mobile hard drives, read-only memories (ROMs), random access memories (RAMs), magnetic disks, or optical disks. It should be noted that, in this document, relational terms such as first and second, etc., are used solely to distinguish one entity or operation from another, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the terms "comprises," "comprising," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising..." does not preclude the presence of additional identical elements in the process, method, article, or apparatus that includes the element. The foregoing description is merely a preferred embodiment of the present application and is not intended to limit the present application. Various modifications and variations are possible for those skilled in the art. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included within the scope of protection of the present application. It should be noted that similar reference numerals and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined or explained in subsequent figures.

[0125] The above description is merely a specific embodiment of the present application, but the scope of protection of the present application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of the present application. Therefore, the scope of protection of the present application should be based on the scope of protection of the claims.

Claims

1. A cluster deployment method, characterized in that: include: Based on user intent, topology analysis is performed using a first neural network, and capacity prediction is performed using a second neural network. Inputting the topology output by the first neural network and the capacity requirement output by the second neural network into an affinity engine to generate a target configuration; Deploy the target cluster according to the target configuration.

2. The method according to claim 1, characterized in that The topological analysis using the first neural network includes: Get hardware communication metrics; Converting the hardware communication metrics into a heterogeneous graph data structure; Inputting the heterogeneous graph data structure into a multi-head graph attention network, and calculating the communication path weight through the multi-head graph attention network; A topology classification result is determined according to the communication path weights.

3. The method according to claim 1, characterized in that The method of using the second neural network to perform capacity prediction includes: Input historical hardware data; Perform forward propagation on the historical hardware data using a long short-term memory network to extract the output of the last time step; Based on the output of the last time step, the capacity demand within the set time is predicted.

4. The method according to claim 3, characterized in that After predicting the capacity demand within a set time period based on the output of the last time step, the capacity prediction using the second neural network further includes: The capacity requirement is subjected to non-negative constraint processing, and the elements of the capacity requirement whose values ​​are less than a preset threshold are reset to a set value; Amplifying the parameters of the dimension specified by the capacity requirement by a preset scaling factor; The capacity demand after non-negative constraint processing and amplification processing is determined to be the actual capacity demand.

5. The method according to any one of claims 1 to 4, characterized in that Inputting the topology structure output by the first neural network and the capacity requirement output by the second neural network into the affinity engine to generate a target configuration includes: Initializing the topology and the capacity requirement; converting the topology and the capacity requirement into an affinity rule for the target cluster; A target configuration is generated according to the affinity rule.

6. The method according to claim 5, characterized in that Before generating the target configuration according to the affinity rule, the method further includes: Mapping each rule in the affinity rules into a node of a conflict graph, and establishing node connection edges according to conflict relationships between the rules to construct the conflict graph; Traversing the conflict graph to determine whether there is a conflicting rule pair in the affinity rules; Generating a target configuration according to the affinity rule includes: When there is no conflicting rule pair in the affinity rules, a target configuration is generated according to the affinity rules.

7. The method according to any one of claims 1 to 4, characterized in that The deploying the target cluster according to the target configuration includes: Generating a topology configuration of the target cluster based on the target configuration through an artificial intelligence algorithm; Parsing the topology configuration through a dynamic template engine to generate a mapping relationship between node type and quantity; According to the mapping relationship, dynamically generate a corresponding deployment strategy; Deploy the target cluster according to the deployment strategy.

8. A cluster deployment device, characterized in that: include: an analysis module, configured to perform topology analysis using a first neural network and capacity prediction using a second neural network based on user intent; a generation module, configured to input the topology structure output by the first neural network and the capacity requirement output by the second neural network into an affinity engine to generate a target configuration; The deployment module is used to deploy the target cluster according to the target configuration.

9. An electronic device, characterized in that: include: A processor and a memory, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the machine-readable instructions are executed by the processor to perform the steps of any one of the methods according to claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the method according to any one of claims 1 to 7.

Images