Cloud disk distribution method
The cloud disk allocation is distributed through deep learning models and the hyperparameter group is updated, which solves the problem of poor cloud disk allocation results in the existing technology and improves load balancing and resource utilization.
Patent Information
- Application Number
- CN202411101271.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-12-03
AI Technical Summary
The existing cloud disk allocation strategy has the problem of poor allocation results, resulting in unbalanced load and low resource utilization.
Deep learning model is used to predict load. By obtaining the ordering information of the target cloud disk, entering the load prediction model to obtain predicted load information, and distributing the cloud disk based on this, updating the hyperparameter group to improve the load prediction effect.
It realizes more suitable cloud disk allocation results, improves load balancing and resource utilization of storage nodes, and solves the problem of poor allocation results.
Smart Images

Figure CN119065606A_ABST
Abstract
Description
Technical Field
[0001] The present application belongs to the field of Internet, and specifically relates to a cloud disk allocation method. Background Art
[0002] As a product of Internet cloud technology, cloud disk is a professional Internet storage tool. Cloud disk can provide information storage, information reading and information download services for enterprises or individuals through the Internet. It is safe, stable and has the characteristics of massive storage, and has been widely used by enterprises and individuals.
[0003] In order to ensure the security and stability of user cloud disk data, a user's cloud disk is usually assigned to a storage node on the distributed storage architecture. In order to ensure the load balancing of each storage node and maximize resource utilization, it is necessary to assign appropriate storage nodes to the cloud disk.
[0004] The cloud disk allocation strategy used in the related art has the problem that the cloud disk allocation result is poor. Summary of the invention
[0005] The embodiment of the present application provides a cloud disk allocation method, which can solve the problem of poor cloud disk allocation results in related technologies.
[0006] In a first aspect, an embodiment of the present application provides a cloud disk allocation method, including:
[0007] Obtain target subscription information of a target cloud disk, where the target cloud disk is the cloud disk applied for;
[0008] The target ordering information is input into a target load prediction model to obtain predicted load information corresponding to the target cloud disk; the target load prediction model is a deep learning model for load prediction, and the target hyperparameter group used by the target load prediction model is obtained based on an initial hyperparameter group and a target update value, and the target update value is used to update the initial hyperparameter group;
[0009] Based on the predicted load information, the target cloud disk is allocated.
[0010] In a second aspect, an embodiment of the present application provides an electronic device, which includes a processor and a memory, wherein the memory stores programs or instructions that can be run on the processor, and when the program or instructions are executed by the processor, the steps of the method described in the first aspect are implemented.
[0011] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, on which a program or instruction is stored, and when the program or instruction is executed, the steps of the method described in the first aspect are implemented.
[0012] In a fourth aspect, an embodiment of the present application provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the method described in the first aspect.
[0013] At least one of the above technical solutions provided in the embodiments of the present application can achieve the following technical effects:
[0014] In an embodiment of the present application, the target subscription information of the target cloud disk is obtained, and the target cloud disk is the cloud disk applied for use; the target subscription information is input into the target load prediction model to obtain the predicted load information corresponding to the target cloud disk; the target load prediction model is a deep learning model for load prediction, and the target hyperparameter group used by the target load prediction model is obtained based on the initial hyperparameter group and the target update value, and the target update value is used to update the initial hyperparameter group; based on the predicted load information, the target cloud disk is allocated. In this way, the predicted load information of the target cloud disk is obtained through the target load prediction model, and the target cloud disk is further allocated through the predicted load information. In this process, the target hyperparameter group used by the cloud disk load prediction model is obtained based on the initial hyperparameter group and the target update value. By updating the initial hyperparameter group with the target update value, a more suitable target hyperparameter group can be selected, so that the target load prediction model has a better load prediction effect, so as to obtain a more suitable cloud disk allocation result, which solves the problem of poor cloud disk allocation results in the related technology. BRIEF DESCRIPTION OF THE DRAWINGS
[0015] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.
[0016] Figure 1 is a flow chart of a cloud disk allocation method provided in an embodiment of the present application;
[0017] Figure 2 is a flow chart of a training process of a target load prediction model provided in an embodiment of the present application;
[0018] Figure 3 It is a specific flow chart of a cloud disk allocation method provided in an embodiment of the present application;
[0019] Figure 4 It is a structural block diagram of a cloud disk allocation device provided in an embodiment of the present application;
[0020] Figure 5It is a structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0021] The following will be combined with the drawings in the embodiments of the present application to clearly describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0022] The terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects, and are not used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable under appropriate circumstances, so that the embodiments of the present application can be implemented in an order other than those illustrated or described here, and the objects distinguished by "first", "second", etc. are generally of one type, and the number of objects is not limited. For example, the first object can be one or more. In addition, "and / or" in the specification and claims represents at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.
[0023] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0024] The cloud disk allocation method provided in the embodiment of the present application is applied to the Internet, and in particular, can be applied to the situation of allocating storage nodes for the cloud disks applied for use. Specifically, the target ordering information of the target cloud disk is input into the target load prediction model to obtain the predicted load information corresponding to the target cloud disk, and then the target cloud disk is allocated according to the predicted load information corresponding to the target cloud disk.
[0025] In the cloud disk allocation method provided in the present application, since the amount of data in the cloud disk is large, a distributed storage architecture can be used to store the cloud disk data, that is, storage nodes for storing cloud disk data are deployed in different regions. Specifically, a cloud disk applied for by a user can be allocated to a storage node on the distributed storage architecture. Cloud disks applied for by different users can be allocated to different storage nodes, and one storage node can store the cloud disk data of multiple users. In addition, the distributed storage architecture may include computing nodes, which may be servers, for completing the scheduling management of cloud disk allocation.
[0026] The cloud disk allocation method provided in the embodiment of the present application can be executed by a target device, wherein the target device can be one electronic device or multiple electronic devices. In the case where the cloud disk allocation method provided in the embodiment of the present application is executed by one electronic device, this electronic device can be a computing node in a distributed storage architecture, can be a terminal device such as a desktop computer, a laptop computer, a mobile phone, a tablet, or a server, such as an independent physical server. In the case where the cloud disk allocation method provided in the embodiment of the present application is executed by multiple electronic devices, these multiple electronic devices can form a service cluster, which cooperate with each other to complete each step. The service cluster can be a server cluster composed of multiple servers, and may include a cloud server capable of cloud computing.
[0027] The cloud disk allocation method provided in the embodiment of the present application is described in detail below through specific embodiments and their application scenarios in conjunction with the accompanying drawings.
[0028] See also Figure 1 , Figure 1 This is a flow chart of a cloud disk allocation method provided in an embodiment of the present application. Figure 1 As shown, the method comprises the following steps:
[0029] Step 110: Obtain target subscription information of a target cloud disk, where the target cloud disk is the cloud disk applied for.
[0030] In the embodiment of the present application, the target cloud disk may be a cloud disk that the target user applies to create for use. When a request from the target user to create a target cloud disk is received, target subscription information of the target cloud disk may be obtained. The target subscription information is account information generated by the user when applying to use the cloud disk. The target subscription information may include at least one of the following information: cloud disk capacity, user attributes of the cloud disk, information about the virtual machine where the cloud disk is located, payment mode of the cloud disk, local disk information of the cloud disk, and snapshot capacity information of the cloud disk.
[0031] Among them, the cloud disk capacity includes the capacity size of the cloud disk to store data, and the user attributes of the cloud disk are used to indicate whether the target user is a VIP. The virtual machine information where the cloud disk is located may include at least one of the number of virtual machine CPU cores (for example, 1 core CPU), virtual machine memory (for example, 1GB memory) and virtual machine type. The virtual machine type is the selection of the storage node allocated to the cloud disk. The payment mode of the cloud disk is the payment method of the target user. The local disk attribute of the cloud disk is used to indicate whether the cloud disk belongs to a local disk (that is, whether the cloud disk is allocated to the storage node in the area where the target user is located). The snapshot capacity information of the cloud disk includes the backup snapshot capacity size of the cloud disk.
[0032] Step 120: Input the target ordering information into the target load prediction model to obtain the predicted load information corresponding to the target cloud disk; the target load prediction model is a deep learning model for load prediction, and the target hyperparameter group used by the target load prediction model is obtained based on the initial hyperparameter group and the target update value, and the target update value is used to update the initial hyperparameter group.
[0033] In the embodiment of the present application, before the target ordering information is input into the target load prediction model, since the target ordering information may include non-numerical information, the target ordering information may be numerated first. For example, the target ordering information includes user attributes of the cloud disk. If the user attributes of the cloud disk indicate that the target user is a VIP, the user attribute information of the cloud disk may be numerated as 1. If the user attributes of the cloud disk indicate that the target user is not a VIP, the user attribute information of the cloud disk may be numerated as 2.
[0034] The target ordering information may include multiple types of information. The data of the same type of information in different target cloud disks is generally different, resulting in different data lengths of the same type of information in different target cloud disks. Therefore, before inputting the target ordering information into the target load prediction model, the length of the target ordering information may be standardized.
[0035] Specifically, the upper limit of the length corresponding to each dimension information in the target subscription information of the target cloud disk can be obtained first. For example, the upper limit of the length of the local disk information of the cloud disk L1, the maximum length of the cloud disk capacity L2, the maximum length of the user attribute of the cloud disk L3, the maximum length of the number of CPU cores of the virtual machine L4, the maximum length of the virtual machine memory L5, the maximum length of the virtual machine type L6, the maximum length of the payment mode of the cloud disk L7, and the maximum length of the snapshot capacity information of the cloud disk L8 can be obtained, and the length of the target subscription information can be determined to be L1+L2+L3+L4+L5+L6+L7+L8.
[0036] Then, the input data for inputting the target ordering information into the target load prediction model can be constructed in the following way: the local disk information of the cloud disk is placed in the first L1 position of the input data, and the insufficient part is filled with zeros; the cloud disk capacity is placed in the L1+1 to L2 positions, and the insufficient part is filled with zeros; the user attributes of the cloud disk are placed in the L2+1 to L3 positions of the input data, and the rest can be deduced in the same way, which will not be repeated here.
[0037] In an embodiment of the present application, since the amount of data of the target ordering information is small, the target load prediction model may be a back propagation (BP) neural network model. The initial hyperparameter group is obtained by initialization of the target load prediction model at the beginning of training, and multiple initial hyperparameter groups may be obtained by multiple initializations. The initial hyperparameter group may be in the form of a vector (e.g., a two-dimensional vector), and an initial hyperparameter group may include multiple hyperparameters (e.g., weights and biases in the target load prediction model).
[0038] In the process of training the target load prediction model, a target update value may be determined, and the initial hyperparameter group may be updated by the target update value, wherein the determination of the target update value is associated with the step size and the random number. Specifically, the initial hyperparameter group may be replaced by the target update value. If the initial hyperparameter group includes multiple hyperparameter groups, a target update value of the hyperparameter group may be determined for each of the multiple hyperparameter groups, and the hyperparameter group may be replaced by the target update value.
[0039] Each time the target load prediction model is trained, a target update value can be determined, that is, each time the target load prediction model can obtain an updated hyperparameter group, through the updated hyperparameter group, the loss value (i.e., error function value) of the target load prediction model when the updated hyperparameter group is applied to the target load prediction model can be determined. The training termination conditions of the target load prediction model include one of the conditions that the loss value is less than a preset threshold and the number of training times reaches a preset maximum number of training times. The target hyperparameter group is the hyperparameter group with the smallest loss value when the target load prediction model training is completed. The target hyperparameter group can be applied to the target load prediction model to realize the prediction of the load information of the target cloud disk by the target load prediction model.
[0040] Step 130: Allocate the target cloud disk based on the predicted load information.
[0041] In an embodiment of the present application, the predicted load information includes at least one of the average number of read / write operations per second (Input / Output Operations Per Second, IOPS) and average bandwidth. The average IOPS can be used to represent the average IOPS value within a preset time period, for example, the average IOPS value of a cloud disk within 30 days. The average storage bandwidth can be used to represent the average storage bandwidth value within a preset time period, for example, the average storage bandwidth value of a cloud disk within 30 days.
[0042] First, obtain the characteristic parameters of the storage node, which include the actual used capacity, actual IOPS, and actual storage bandwidth. At the same time, obtain the capacity upper limit, IOPS upper limit, and actual storage bandwidth upper limit of the storage node. The number of the storage nodes may be multiple, and for any one of the multiple storage nodes, based on the capacity upper limit, IOPS upper limit, actual storage bandwidth upper limit, and characteristic parameters of the storage node, the actual used capacity, actual IOPS, and actual storage bandwidth resource utilization of the storage node can be obtained. For details, refer to the following formula:
[0043]
[0044] Among them, η cl The resource utilization rate used to represent the actual usage capacity of the storage node, η hl It is used to represent the actual IOPS resource utilization of the storage node, η gl Used to represent the resource utilization of the actual storage bandwidth of the storage node. c1 is the actual used capacity of the storage node, h1 is the actual IOPS of the storage node, g1 is the actual storage bandwidth of the storage node, C is the storage capacity upper limit of the storage node, H is the IOPS upper limit of the storage node, and G is the actual bandwidth upper limit of the storage node.
[0045] By determining the actual used capacity, actual IOPS, and actual storage bandwidth resource utilization of all storage nodes, the current usage of all storage nodes can be obtained (before allocating the target cloud disk).
[0046] Then, the capacity of the target cloud disk can be obtained from the target subscription information, and combined with the predicted load information to obtain the characteristic parameters of the target cloud disk, the characteristic parameters of the target cloud disk include the capacity of the target cloud disk, the average IOPS of the target cloud disk, and the average bandwidth of the target cloud disk. For any storage node among the multiple storage nodes, based on the characteristic parameters of the target cloud disk, the resource utilization rate of the actual used capacity of the target cloud disk added to the storage node, the resource utilization rate of the actual IOPS, and the resource utilization rate of the actual storage bandwidth can be determined by the following formula.
[0047]
[0048] Among them, η' cl It is used to represent the resource utilization rate of the actual used capacity of the target cloud disk added to the storage node, η' hl It is used to represent the actual IOPS resource utilization of the target cloud disk added to the storage node, η' glIt is used to indicate the resource utilization rate of the actual storage bandwidth of the target cloud disk added to the storage node. c1 is the actual used capacity of the storage node, h1 is the actual IOPS of the storage node, g1 is the actual storage bandwidth of the storage node, c2 is the capacity of the target cloud disk, h2 is the average IOPS of the target cloud disk, g2 is the average bandwidth of the target cloud disk, C is the upper limit of the storage capacity of the storage node, H is the upper limit of the IOPS of the storage node, and G is the upper limit of the actual bandwidth of the storage node.
[0049] For any storage node among the multiple storage nodes, after determining the resource utilization of the actual used capacity of the target cloud disk added to the storage node, the resource utilization of the actual IOPS, and the resource utilization of the actual storage bandwidth, the mean of the three resource utilizations of the storage node can be calculated by the following formula, and the resource utilization vector (η') of the storage node can be constructed. cl ,η' cl ,η' cl ) and the resource utilization mean vector
[0050]
[0051] in, Add the average of the three resource utilizations of the storage node to the target cloud disk.
[0052] After constructing the resource utilization vector and the resource utilization mean vector of the storage node, the resource utilization vector (η') of the target cloud disk added to the storage node can be determined by the following formula: cl ,η' cl ,η' cl ) and the mean vector of resource utilization The Manhattan distance between:
[0053]
[0054] Among them, η' cl It is used to represent the resource utilization rate of the actual used capacity of the target cloud disk added to the storage node, η' hl It is used to represent the actual IOPS resource utilization of the target cloud disk added to the storage node, η' gl It is used to indicate the resource utilization rate of the actual storage bandwidth of the target cloud disk added to the storage node. is the mean of the three resource utilizations of the target cloud disk added to the storage node, L is the resource utilization vector (η' cl ,η' cl ,η' cl ) and the resource utilization mean vector The Manhattan distance between .
[0055] For multiple storage nodes, the Manhattan distances corresponding to the multiple storage nodes can be determined in the above manner. If the Manhattan distance corresponding to a storage node is small, it can be considered that the load of the three resource utilization rates of the storage node is relatively balanced. In other words, the target cloud disk can be allocated to the storage node with the smallest Manhattan distance.
[0056] In an embodiment of the present application, the target subscription information of the target cloud disk is obtained, and the target cloud disk is the cloud disk applied for use; the target subscription information is input into the target load prediction model to obtain the predicted load information corresponding to the target cloud disk; the target load prediction model is a deep learning model for load prediction, and the target hyperparameter group used by the target load prediction model is obtained based on the initial hyperparameter group and the target update value, and the target update value is used to update the initial hyperparameter group; based on the predicted load information, the target cloud disk is allocated. In this way, the predicted load information of the target cloud disk is obtained through the target load prediction model, and the target cloud disk is further allocated through the predicted load information. In this process, the target hyperparameter group used by the cloud disk load prediction model is obtained based on the initial hyperparameter group and the target update value. By updating the initial hyperparameter group with the target update value, a more suitable target hyperparameter group can be selected, so that the target load prediction model has a better load prediction effect, so as to obtain a more suitable cloud disk allocation result, which solves the problem of poor cloud disk allocation results in the related technology.
[0057] The present application embodiment provides a method for training the above target load prediction model. Figure 2 , Figure 2 FIG. 1 is a flow chart of a training process of a target load prediction model provided by an embodiment of the present application. Figure 2 As shown, the training process of the target load prediction model includes the following steps:
[0058] Step 210: Obtain a training data set and an initial hyperparameter set.
[0059] In the embodiment of the present application, the training data set includes cloud disk historical subscription information and cloud disk historical load information, and there is a one-to-one correspondence between the cloud disk historical subscription information and the cloud disk historical load information to construct a training data pair. The cloud disk historical subscription information can be used as input data of the target load prediction model, and the cloud disk historical load information can be used as the load true value label of the corresponding cloud disk historical subscription information, which is used to obtain the loss value of the target load prediction model during the training process of the target load prediction model.
[0060] In an embodiment of the present application, the cloud disk historical subscription information may include at least one of the following information: cloud disk capacity, user attributes of the cloud disk, information about the virtual machine where the cloud disk is located, payment mode of the cloud disk, local disk information of the cloud disk, and snapshot capacity information of the cloud disk. It should be noted that the information contained in the cloud disk historical subscription information may be consistent with the information category contained in the target subscription information of the target cloud disk to determine the prediction effect of the target load prediction model. At the same time, the cloud disk historical subscription information may be processed for data length standardization. The specific processing method may refer to the method for processing the data length standardization of the target subscription information in step 120, which will not be described in detail here.
[0061] In one embodiment of the present application, before training the load prediction model to be trained, the method also includes: clustering the cloud disk historical ordering information to obtain N clustering results; for the kth clustering result among the N clustering results, obtaining target load information, the target load information including the cloud disk historical load information corresponding to each cloud disk historical ordering information in the kth clustering result; obtaining the mean of the target load information; determining the mean of the target load information as the actual load information corresponding to all cloud disk historical ordering information in the kth clustering result; and using the actual load information as the true value label of all cloud disk historical ordering information in the kth clustering result.
[0062] Since the load prediction model to be trained can be a classification prediction model, the data of the training data set can be processed first to obtain load information of different categories. When the load prediction model to be trained is trained using the data of the training data set, the classification load prediction effect of the load prediction model to be trained can be improved.
[0063] In an embodiment of the present application, the cloud disk historical ordering information may include cloud disk historical ordering information corresponding to B users, and the B cloud disk historical ordering information may be clustered to obtain N clustering results, wherein B and N are both positive integers, and B is greater than N. It should be noted that the above-mentioned B cloud disk historical ordering information can all be in the form of vectors, for example, z = (z1, z2, z3, z4, z5, z6, z7, z8), and z1~z8 can be 8 types of information after numerical processing (cloud disk capacity, user attributes of cloud disk, number of virtual machine CPU cores, virtual machine memory, virtual machine type, cloud disk payment mode, cloud disk local disk information, and cloud disk snapshot capacity information). Specifically, the B cloud disk historical ordering information can be clustered in the following way:
[0064] First, the number of clustering results N (for example, 8) can be preset. The larger N is, the more refined the clustering of the cloud disk historical ordering information is. N cloud disk historical ordering information is randomly obtained from the B cloud disk historical ordering information, and the N cloud disk historical ordering information is used as the clustering center. For each cloud disk historical ordering information except the N clustering centers in the B cloud disk historical ordering information, the Euclidean distance between the cloud disk historical ordering information and the N clustering centers is determined, and the cloud disk historical ordering information can be assigned to the clustering center with the smallest Euclidean distance. After allocating each cloud disk historical ordering information except the N clustering centers in the B cloud disk historical ordering information, N clustering results of the first clustering can be obtained, and each of the N clustering results can include one or more cloud disk historical ordering information.
[0065] Then, the N clustering results obtained by the first clustering can be updated, for example, the cluster center of each clustering result can be adjusted. Specifically, for any one of the N clustering results, all cloud disk historical ordering information contained in the clustering result is averaged, and the result obtained by the average is determined as the updated cluster center to obtain the updated N cluster centers. After obtaining the N updated cluster centers, the B cloud disk historical ordering information can be re-clustered in the same manner as the first clustering to obtain the N clustering results obtained by the second clustering. And so on, iterative clustering can be performed, which will not be described in detail later.
[0066] The termination condition for iterative clustering may include the convergence of N cluster centers, that is, the change in the N cluster centers of multiple clustering is less than a preset threshold, and the change may be the Euclidean distance between the cluster centers of the two clusterings. When the termination condition for iterative clustering is met, it can be determined that the clustering of all cloud disk historical ordering information is completed, and N clustering results of clustering completion are obtained.
[0067] For each of the N clustering results, the cloud disk historical load information corresponding to the cloud disk historical ordering information in the clustering result may be averaged, and the average may be determined as the average cloud disk historical load information corresponding to the clustering result. For each of the B cloud disk historical ordering information, the average cloud disk historical load information corresponding to the clustering result to which the cloud disk historical ordering information belongs may be determined as the actual load information of the cloud disk historical ordering information, and the actual load information may be the true value label of the load information of the cloud disk historical ordering information.
[0068] In an embodiment of the present application, by clustering the training data set, the load prediction model to be trained can be trained using the training data set after clustering, which can improve the training effect of the load prediction model to be trained.
[0069] Step 220: In the process of training the load prediction model to be trained based on the training data set, multiple loss values obtained by the load prediction model to be trained using multiple different hyperparameter groups are obtained; the multiple different hyperparameter groups are obtained based on multiple update values.
[0070] In an embodiment of the present application, multiple different hyperparameter groups are applied to the load prediction model to be trained, and multiple loss values can be obtained based on the training data set. Specifically, a certain cloud disk historical ordering information in the training data set is input into the load prediction model to be trained to obtain the load prediction result of the cloud disk historical ordering information, and the loss value of the load prediction model to be trained can be determined based on the load prediction result and the actual load information of the cloud disk historical ordering information. The multiple loss values can be used to indicate that applying different hyperparameter groups to the load prediction model to be trained can have uneven prediction effects.
[0071] In one embodiment of the present application, the multiple update values include at least one update value among a first update value, a second update value, a third update value and a fourth update value; the first update value is obtained based on the first step length, the second update value is obtained based on the second step length, the third update value is obtained based on the third step length, the fourth update value is obtained based on the fourth step length, and the second step length is the same as the third step length; the multiple different hyperparameter groups include at least one hyperparameter group among a first hyperparameter group obtained based on the first update value, a second hyperparameter group obtained based on the second update value, a third hyperparameter group obtained based on the third update value, and a fourth hyperparameter group obtained based on the fourth update value.
[0072] In an embodiment of the present application, the first step length, the second step length, the third step length and the fourth step length can all be determined based on the number of training times of the current load prediction model to be trained. During the determination process, the second step length and the third step length can be the same. The first step length, the second step length and the fourth step length can consider introducing random numbers in different ways, so that the determination of the first step length, the second step length and the fourth step length is more random, so as to avoid falling into a local optimum in the process of determining the first update value, the second update value and the fourth update value.
[0073] It should be noted that the determination of the first update value, the second update value, the third update value, and the fourth update value can be performed independently of each other in a multi-threaded manner. During the execution process, the first update value, the second update value, the third update value, and the fourth update value can be determined according to a preset logical order. For example, the first update value is determined first, and the determination of the second update value is triggered after the first update value is determined, and the determination of the third update value is triggered after the second update value is determined, and the determination of the fourth update value is triggered after the third update value is determined. The following will be explained from the determination method of the first update value, the second update value, the third update value, and the fourth update value.
[0074] In a first aspect, by performing a neighborhood search on the hyperparameter group, a better position can be determined from around the hyperparameter group to determine the first update value.
[0075] Since the idea of determining the first update value is neighborhood search, in order to conduct a wider neighborhood search for the location of the hyperparameter group in the solution space, a larger step size can be determined in the early stage of training the load prediction model to be trained to expand the scope of the neighborhood search. In the later stage of training the load prediction model to be trained, a smaller step size can be determined to increase the convergence speed of the loss value of the load prediction model to be trained. Therefore, the first step size in the first update value can be decreased according to the current number of training times.
[0076] For example, to obtain the number of training times for the current load prediction model to be trained, the first step length can be determined by the following formula:
[0077]
[0078] Among them, step max is the preset upper limit of the first step length, step min is the preset lower limit of the first step length, t is the current training times of the load prediction model to be trained, T is the maximum training times, and step1 is the first step length. It should be noted that the above method for determining the first step length is only an example, and other methods can be used to obtain the first step length, such as
[0079] The load prediction model to be trained can currently use M hyperparameter groups. After determining the first step length, the first update value can be determined in the following manner:
[0080] X i1 =X i +step1(2rand1-1);
[0081] Among them, X i is the i-th hyperparameter group currently used by the load prediction model to be trained, X i1is the first updated value of the i-th hyperparameter group, step1 is the first step length, and the first step length is obtained based on the current number of training times, and rand1 is used to represent a random number between (-1,1).
[0082] Secondly, by analyzing the degree of clustering between the hyperparameter groups, the second update value is determined.
[0083] In an embodiment of the present application, the initial hyperparameter group includes M initial hyperparameter groups in vector form (e.g., two-dimensional vectors), the multiple update values include a second update value, and the target update value is obtained based on the second update value. For the i-th initial hyperparameter group among the M initial hyperparameter groups, a first hyperparameter group corresponding to the i-th initial hyperparameter group is determined from the M initial hyperparameter groups, the first hyperparameter group being a hyperparameter group whose Euclidean distance with the i-th initial hyperparameter group is less than a set threshold; based on the first hyperparameter group, a central hyperparameter group corresponding to the i-th initial hyperparameter group is determined; a search factor and a second step length are obtained, the search factor and the second step length are both determined according to the current number of training times; when the search factor and the central hyperparameter group meet the first condition, based on the second step length and the central hyperparameter group, a second update value corresponding to the i-th hyperparameter group is determined.
[0084] In the embodiment of the present application, the second updated values of the M initial hyperparameter groups can be determined for the M initial hyperparameter groups, and the second updated values of the M updated hyperparameter groups can also be determined for the M hyperparameter groups updated during the training process. The following is an example of determining the second updated values of the M initial hyperparameter groups for a specific introduction.
[0085] For the i-th hyperparameter group among the M initial hyperparameter groups, the i-th hyperparameter group X can be obtained i and other hyperparameter groups (the hyperparameter groups except the i-th hyperparameter group in the M initial hyperparameter groups, i.e., X i ≠X j )X j The Euclidean distance between ||X i -X j ||, and determine the hyperparameter group whose Euclidean distance among the other hyperparameter groups is less than the set threshold as the first target hyperparameter group corresponding to the i-th hyperparameter group. The set threshold can be updated according to the current training times of the load prediction model to be trained. Specifically, the initial value of the set threshold can be set and the set threshold can be updated in the following manner:
[0086]
[0087] Among them, S0 is the initial value of the set threshold, S tis the threshold used in the tth training process, ub d Indicates the upper limit of the d-th dimension parameter in the hyperparameter group, lb d Indicates the lower limit of the d-th dimension parameter in the hyperparameter group. The hyperparameter group may be a two-dimensional vector, and the d-th dimension parameter may be a one-dimensional vector in the two-dimensional vector. For example, the hyperparameter group may be (a, b, c), where a, b and c are all one-dimensional vectors, and the second dimension parameter of the hyperparameter is the one-dimensional vector b. The upper limit of the second dimension parameter is the length of the one-dimensional vector b, and the lower limit may be 1. a1 is a constant less than 0, which is used to adjust the speed of dynamic reduction of the set threshold. The smaller a1 is, the faster the threshold reduction speed is. It can be set according to actual needs. S min To set the lower limit of the threshold, it is used to avoid setting the threshold to 0. For example, you can set S min =0.001.
[0088] After obtaining the first target hyperparameter group corresponding to the i-th hyperparameter group, the first target hyperparameter group may include NF hyperparameter groups, and the mean of the NF hyperparameter groups may be determined by the following formula to obtain the central hyperparameter group corresponding to the first target hyperparameter group:
[0089]
[0090] Among them, X c is the central hyperparameter group, NF is the number of hyperparameter groups included in the first target hyperparameter group, X' f is the fth hyperparameter group in the first target hyperparameter group (NF hyperparameter groups).
[0091] After obtaining the central hyperparameter group, the loss value of the load prediction model to be trained when the central hyperparameter group is applied to the load prediction model to be trained, and the loss value of the load prediction model to be trained when the i-th hyperparameter group is applied to the load prediction model to be trained can be determined. At the same time, a search factor can be obtained, and the search factor is used to determine whether to update the i-th hyperparameter group. Specifically, whether to update the i-th hyperparameter group can be determined by judging whether the following first condition is satisfied:
[0092]
[0093] Among them, E c is the center hyperparameter X c The error function group, E i is the i-th hyperparameter group X i The loss value of , δ is the search factor, and NF is the number of hyperparameter groups included in the first target hyperparameter group.
[0094] The setting of the search factor can be updated according to the current training times of the load prediction model to be trained. In the early stage of model training, a smaller search factor can be determined so that all hyperparameter groups are in a divergent state to expand the search range; in the later stage of model training, a larger search factor can be determined so that more hyperparameter groups are close to the extreme value point. Specifically, the search factor can be determined according to the following methods:
[0095]
[0096] Among them, δ is the search factor, δ max is the preset upper limit of the search factor, δ min is the preset lower limit of the search factor, t is the current training times of the load prediction model to be trained, and T is the preset maximum training times.
[0097] When the above-mentioned first condition is met, a second step size can be obtained, and the second step size can be set to the product of a set threshold and a random coefficient (for example, a random number between 0.25-0.6), and the second update value corresponding to the i-th hyperparameter group is determined based on the second step size and the central hyperparameter group.
[0098] In one embodiment of the present application, the second update value corresponding to the i-th hyperparameter group is determined by the following formula:
[0099]
[0100] Among them, X i is the i-th hyperparameter group, step2 is the second step length, rand2 is used to represent a random number between (0,1), X c is the central hyperparameter group, X i2 is the second update value corresponding to the ith hyperparameter group. The above formula can be used to introduce a random number in determining the second update value to avoid falling into a local optimum during the process of updating the hyperparameter group.
[0101] Thirdly, approach the local optimal value and implement local fine search to determine the third update value.
[0102] Specifically, for the i-th hyperparameter group among the M initial hyperparameter groups, the first hyperparameter group corresponding to the i-th hyperparameter group can be obtained. The method for determining the first hyperparameter group corresponding to the i-th hyperparameter group here can refer to the second update value, which will not be repeated here. The first hyperparameter group may include multiple hyperparameter groups, and the loss values when the multiple hyperparameter groups are applied to the load prediction model to be trained can be obtained, and the hyperparameter group with the smallest loss value in the first hyperparameter group is determined as the second hyperparameter group corresponding to the i-th hyperparameter group, so as to achieve the local optimal value of the i-th hyperparameter group close to complete the local fine search.
[0103] Then, a third step length is obtained, which is determined according to the current number of training times. The third step length is consistent with the second step length, and the determination method of the second step length can be referred to. The third update value of the i-th hyperparameter group is determined by the following formula:
[0104]
[0105] Among them, X i3 is the third update value of the i-th hyperparameter group, X i is the i-th hyperparameter group, rand2 is used to represent a random number between (0,1), step3 is used to represent the third step, and X max is the second hyperparameter group. The above formula can be used to approach the local optimal value of the i-th hyperparameter group to complete the local refined search.
[0106] Fourthly, a target dimension parameter in the hyperparameter group is selected for random update to obtain a fourth updated value.
[0107] Specifically, determine the target dimension parameter of the initial hyperparameter group; based on the current number of training times, obtain a fourth step length; based on the fourth step length, update the target dimension parameter by the following formula to obtain an updated target dimension parameter: i4 =x id +step4*rand1; based on the updated target dimension parameter, obtain the fourth updated value; where x id is the target dimension parameter, x i4 is the updated target dimension parameter, step4 is the fourth step length, and rand1 is used to represent a random number between (-1, 1).
[0108] In the embodiment of the present application, the initial hyperparameter group may be a two-dimensional vector, such as (a, b, c), where a, b and c are all one-dimensional vectors, and a parameter of one dimension may be randomly selected from the two-dimensional vector, and the parameter may be determined as the target dimension parameter, for example, the target dimension parameter may be a one-dimensional vector c. Then the fourth step length may be obtained, and the fourth step length may be set to step1*rand2, where rand2 is used to represent a random number between (0,1).
[0109] Through the above formula, the target dimension parameter can be randomly updated to obtain an updated target dimension parameter, and the original target dimension parameter can be replaced with the updated target dimension parameter to determine the fourth update value. For example, if the one-dimensional vector c is updated to obtain a one-dimensional vector c', the fourth update value can be determined to be (a, b, c').
[0110] It should be noted that for each dimension parameter in the initial hyperparameter group, there is a range of values for the parameter, that is, the value of the parameter is meaningful only if it falls within the range. Therefore, after updating the target dimension parameter, it can be determined whether the updated target dimension parameter value is meaningful. If the updated target dimension parameter is meaningless, the updated target dimension parameter can be processed as out of bounds, such as not updating the target dimension parameter or re-updating the target dimension parameter.
[0111] The fourth update value is determined in the above manner, and the target dimension parameter is randomly selected from the initial parameter group for updating. The random update method helps to escape the local optimum in the process of updating the hyperparameter group, so as to improve the effect of training the load prediction model to be trained.
[0112] Step 230: Determine the hyperparameter group corresponding to the minimum loss value among the multiple loss values as the target hyperparameter group.
[0113] In an embodiment of the present application, the initial hyperparameter group may include multiple hyperparameter groups. For any one of the multiple hyperparameter groups, a first update value, a second update value, a third update value, and a fourth update value may be determined. The loss values when the four update values are applied to the load prediction model to be trained may be obtained, and the update value with the smallest loss value is determined as the final update value of the hyperparameter group, and the final update value is determined as the updated hyperparameter group. After the load prediction model to be trained stops iterative updating, the updated hyperparameter group of the multiple hyperparameter groups may be determined. When the multiple updated hyperparameter groups are applied to the load prediction model to be trained, multiple loss values of the load prediction model to be trained may be obtained, and the hyperparameter group corresponding to the smallest loss value among the multiple loss values may be determined as the target hyperparameter group.
[0114] In one embodiment of the present application, after determining the hyperparameter group corresponding to the minimum loss value among the multiple loss values as the target hyperparameter group, the method further includes: in the process of training the load prediction model to be trained, if the loss value of at least Q training times has not decreased, performing reverse learning and Cauchy mutation on the target hyperparameter group; wherein Q is a positive integer greater than 1.
[0115] In the embodiment of the present application, during the training process of the load prediction model to be trained, the update of the hyperparameter group used by the model may cause the loss value of the model to decrease, or may cause the loss value of the model to increase. If the loss value of the model does not decrease after multiple trainings, it can be considered that the model may fall into a local optimum, and continuing to update the hyperparameter group will not improve the training effect of the model much.
[0116] Therefore, during the training of the load prediction model to be trained, the optimization effect of the load prediction model to be trained can be observed, and the optimization effect includes that the loss value of at least Q training times has not decreased, and the loss value of Q training times has decreased. If it is observed that the loss value of at least Q training times has not decreased, reverse learning and Cauchy mutation can be performed on the hyperparameter group currently used by the prediction model to be trained to jump out of the local optimum.
[0117] Step 240: Based on the target hyperparameter group, obtain the target load prediction model.
[0118] In an embodiment of the present application, after obtaining the target hyperparameter group, the target hyperparameter group can be directly applied to the load prediction model to be trained to complete the deployment of the target load prediction model.
[0119] In an embodiment of the present application, a training process for a load prediction model to be trained is provided, and multiple different hyperparameter values are obtained through multiple update values to achieve a refined search for hyperparameter groups, thereby achieving the selection of the optimal hyperparameter group, thereby ensuring that the target load prediction model has a better load prediction effect.
[0120] Please refer to the following specific embodiment, which is used to further describe the cloud disk allocation method provided in the embodiment of the present application, including the following steps:
[0121] Step 1: Cluster historical information to obtain basic load balancing information;
[0122] In the embodiment of the present application, the above step 1 may include the following steps 1.1-1.8:
[0123] 1.1. Collect historical information of the allocated cloud disk, which includes the cloud disk subscription information and cloud disk historical load information corresponding to the user (i.e. the cloud disk historical subscription information and cloud disk historical load information mentioned above).
[0124] Among them, the cloud disk subscription information is used to indicate the account information generated when the user creates a new cloud disk, which may include the VIP category, the number of CPU cores of the user's virtual machine, the memory size of the user's virtual machine, the user's payment method, whether the cloud disk is a local disk, the selection of the cloud server corresponding to the cloud disk, the snapshot size, and the cloud disk capacity. It is worth noting that only a few of the above data may be included, not all. The cloud disk historical load information is used to indicate the user's average bandwidth and average IOPS in a certain historical time period (such as 10 days, 20 days, 30 days, etc.).
[0125] 1.2. Digitalize all cloud disk subscription information.
[0126] 1.3. Based on the pre-set number of clusters K, randomly sample K cluster centers from the cloud disk subscription information of all users.
[0127] 1.4. For the cloud disk subscription information corresponding to other users except the cluster center, obtain the Euclidean distance between the cloud disk subscription information and the K cluster centers, and assign the cloud disk subscription information to the category corresponding to the cluster center with the shortest Euclidean distance.
[0128] 1.5. After classifying the cloud disk subscription information of all users, update the cluster center corresponding to each category. The specific updating method is: take the mean of all cloud disk subscription information in each category as the new cluster center.
[0129] 1.6. Determine whether the cluster centers converge, that is, whether the multiple changes of the cluster centers of each category are less than the set threshold. If so, clustering is completed, otherwise, return to the step of clustering the cloud disk subscription information.
[0130] 1.7. After clustering is completed, the cloud disk historical load information corresponding to the cloud disk subscription information in each category is averaged to obtain the average cloud disk historical load information of each category.
[0131] 1.8. For each cloud disk subscription information in each category, the average cloud disk historical load information corresponding to the category is used as the target cloud disk historical load information (i.e. the actual load information mentioned above), so as to obtain the basic load balancing information.
[0132] Step 2: Construction, training and deployment of the intelligent prediction model (i.e. the target load prediction model mentioned above).
[0133] In the embodiment of the present application, the above step 2 may include the following steps 2.1-2.10:
[0134] 2.1. Since the amount of cloud disk ordering information is small, the BP neural network can be used to build an intelligent prediction model.
[0135] 2.2. Use the cloud disk subscription information and the target cloud disk historical load information corresponding to the cloud disk subscription information to construct a training data pair, where the cloud disk subscription information is used as input data and the target cloud disk historical load information corresponding to the cloud disk subscription information is used as the true value label. In this way, the loss value corresponding to the intelligent prediction model can be obtained during the training process.
[0136] The cloud disk subscription information includes VIP category, the number of CPU cores of the user's virtual machine, the memory size of the user's virtual machine, the user's payment method, whether the cloud disk is a local disk, the selection of the cloud server corresponding to the cloud disk, the snapshot size, and the cloud disk capacity. Different data may result in different data lengths. Therefore, you can first obtain the maximum length corresponding to each dimension of data, that is, the maximum length L1 of the VIP category, the maximum length L2 of the number of CPU cores of the user's virtual machine, the maximum length L3 of the memory size of the user's virtual machine, the maximum length L4 of the user's payment method, the maximum length L5 of whether the cloud disk is a local disk, the maximum length L6 of the selection of the cloud server corresponding to the cloud disk, the maximum length L7 of the snapshot size, and the maximum length L8 of the cloud disk capacity.
[0137] Therefore, the input data can be constructed as follows: the VIP category is placed in the first L1 bit of the input data, and the insufficient bits are filled with zeros; the number of CPU cores of the user virtual machine is placed in the L1+1 to L2 bits, and the insufficient bits are filled with zeros; the rest will not be repeated, and the same can be said for the rest.
[0138] 2.3. Initialize the hyperparameters of the intelligent prediction model and obtain the hyperparameter group corresponding to the intelligent prediction model; in order to facilitate subsequent multi-threaded optimization, this application initializes the hyperparameters of the intelligent prediction model multiple times to obtain M hyperparameter groups corresponding to the intelligent prediction model. Among them, the hyperparameters represent the weights and biases corresponding to the intelligent prediction model, and each hyperparameter group can be regarded as a vector.
[0139] 2.4. For each hyperparameter group, perform a neighborhood search on the hyperparameter group to detect a better position around it and obtain the first update value corresponding to the hyperparameter group, specifically:
[0140] X i1 =X i +step1(2rand1-1)
[0141]
[0142] Among them, X i1 is the first updated value, X i is the hyperparameter group currently used by the load prediction model to be trained, step1 is the first step length, which is obtained based on the current number of training times, rand1 is used to represent a random number between (-1, 1), step max is the preset upper limit of the first step length, step minis the preset lower limit of the first step length, t is the current training number of the load prediction model to be trained, and T represents the maximum number of training times. This application is aimed at neighborhood search, and a wider field search can be performed from the location of the hyperparameter group in the solution space, so it is not easy to miss the optimal solution. At the same time, a larger step size range can be selected in the early stage to expand the search range, and a smaller step size can be selected in the later stage to increase the convergence speed, so the step size should be gradually reduced.
[0143] 2.5. For each hyperparameter group, the degree of aggregation is used as a basis to obtain the second update value corresponding to the hyperparameter group.
[0144] Specifically:
[0145] Get the i-th hyperparameter group X i With other hyperparameter groups X j The distance between them is: ||X i -X j ||, and set the distance ||X i -X j ||Other hyperparameter groups that are smaller than the set threshold are taken as the first target hyperparameter group (i.e., the first hyperparameter group mentioned above), X i ≠X j .
[0146] Among them, the initial value of the threshold is set to: ub d Indicates the upper limit of the d-th dimension parameter in the hyperparameter group, lb d It represents the lower limit of the d-th dimension parameter in the hyperparameter group, and D represents the total number of dimensions. As the iteration proceeds, the threshold is updated once, specifically: S t represents the set threshold used in the tth training; a1 represents a constant less than 0, which is used to adjust the speed of dynamic reduction of the threshold. The smaller its value, the faster the threshold decreases, so it can be set according to actual needs; S min =0.001.
[0147] Get the mean of each parameter in all first target hyperparameter groups, and get the central hyperparameter group corresponding to the first target hyperparameter group:
[0148]
[0149] Among them, X c is the central hyperparameter group, NF is the number of hyperparameter groups included in the first target hyperparameter group, X' f is the fth hyperparameter group in the first target hyperparameter group (NF hyperparameter groups).
[0150] Get the i-th hyperparameter group X i The loss value E iand the central hyperparameter group X c The loss value E c , and judge E c Is it less than E? i ,and Is it true? If so, get the second update value corresponding to the hyperparameter group as: Otherwise, the second updated value corresponding to the hyperparameter group is obtained as its original value.
[0151] Among them, δ is the search factor, δ max is the preset upper limit of the search factor, δ min is the preset lower limit of the search factor. In the early stage of this application, a smaller search factor is set to make all hyperparameter groups diverge and expand the search range. In the later stage, a larger search factor is set to make more hyperparameter groups approach the extreme point. i2 is the second update value corresponding to the i-th hyperparameter group, rand2 is used to represent a random number between (0,1), step2 is the second step size, and the second step size can be set to the product of a set threshold and a random coefficient (for example, a random number between 0.25-0.6).
[0152] 2.6. To set the threshold S t As a constraint, get the value of the i-th hyperparameter group X i Distance S t The optimal hyperparameter set X in max (The optimal value refers to the minimum corresponding loss value, that is, the second hyperparameter group mentioned above), and based on the optimal hyperparameter group X max Get the third updated value.
[0153] The third update value is specifically:
[0154]
[0155] Among them, X i3 is the third update value of the i-th hyperparameter group, X i is the i-th hyperparameter group, rand2 is used to represent a random number between (0,1), step2 is used to represent the second step length, and X max is the second hyperparameter group. The second step length here is equivalent to the third step length used in calculating the third update value above. Since the step lengths used in calculating the second update value and the third update value are the same, the same identifier is used here. Through the above steps of obtaining the third update value, it is possible to further approach the local optimal value, thereby achieving local fine search.
[0156] 2.7. For the i-th hyperparameter group X i, randomly select a parameter of one dimension to obtain a target dimension parameter; for the target dimension parameter, randomly move forward to obtain the fourth update value.
[0157] For this target dimension parameter, the random advance can be:
[0158] x i4 =x id +step3*rand1
[0159] Among them, x id is the target dimension parameter, x i4 is the updated target dimension parameter, step3 is the third step length, and rand1 is used to represent a random number between (-1, 1). Random execution can help to jump out of the local optimum and achieve better training results.
[0160] It is worth mentioning that after each update, out-of-bounds processing is required.
[0161] 2.8. Obtain the update value with the smallest loss value among the first update value, the second update value, the third update value, and the fourth update value, and use it as the final update value of the i-th hyperparameter group. Based on the final update value, use a greedy strategy to update the i-th hyperparameter group.
[0162] 2.9. Obtain the hyperparameter group with the smallest loss value, obtain the worst hyperparameter group, and determine the optimization effect of the loss value of the worst hyperparameter group; when the optimization effect is that the loss value has not decreased for at least N rounds, perform reverse learning and Cauchy mutation on the worst hyperparameter group.
[0163] The optimization effect includes that the loss value does not decrease in at least N rounds or the loss value decreases in N rounds,
[0164] 2.10. Obtain the loss values of all hyperparameter groups and determine the minimum loss value; when the minimum loss value meets the iteration end condition or the current training times reaches the maximum training times, apply the hyperparameter group corresponding to the minimum loss value (that is, the target hyperparameter group mentioned above) to the intelligent prediction model to realize the deployment of the intelligent prediction model.
[0165] If the end condition is not met, the process returns to 2.4.
[0166] Through the above training and deployment methods, the intelligent prediction model can be trained relatively quickly without falling into the local optimum. It can also perform local refined search to obtain the optimal hyperparameter group, thereby ultimately ensuring that the intelligent prediction model has a better recognition effect.
[0167] Optionally, after the training is completed, the performance parameters of the intelligent prediction model can also be measured. When the performance parameters do not meet the standards, retraining is performed to ensure the actual application effect.
[0168] Step 3: Obtain the target subscription information corresponding to the new cloud disk created by the user (i.e., the target cloud disk mentioned above), convert the target subscription information in the form of input data of the intelligent prediction model, and input the converted target subscription information into the intelligent prediction model to obtain the predicted load information corresponding to the target subscription information. The predicted load information may include average bandwidth and average IOPS.
[0169] It is worth noting that in order to reduce the amount of data and speed up the recognition speed, the data input into the intelligent prediction model can be normalized.
[0170] Step 4: Obtain the capacity of the new cloud disk created by the user, and combine it with the predicted average bandwidth and average IOPS to obtain the characteristic parameters of the new cloud disk;
[0171] Step 5: Obtain the actual used capacity, actual IOPS, and actual bandwidth of each storage node in the distributed storage architecture to obtain characteristic parameters of the storage node.
[0172] Step 6: According to the characteristic parameters of the storage node, obtain the resource utilization corresponding to the actual used capacity, actual IOPS and actual bandwidth of each storage node.
[0173] The resource utilization rates corresponding to the actual used capacity, actual IOPS, and actual bandwidth can be:
[0174]
[0175] Among them, η cl The resource utilization rate used to represent the actual used capacity of the storage node, η hl It is used to represent the actual IOPS resource utilization of the storage node, η gl It is used to indicate the resource utilization of the actual storage bandwidth of the storage node. c1 is the actual used capacity of the storage node, h1 is the actual IOPS of the storage node, g1 is the actual storage bandwidth of the storage node, C is the upper limit of the storage capacity of the storage node, H is the upper limit of the IOPS of the storage node, and G is the upper limit of the actual bandwidth of the storage node.
[0176] Step 7: According to the characteristic parameters of the new cloud disk, obtain the resource utilization rate of the storage node after the new cloud disk is added to the storage node.
[0177] The resource utilization is:
[0178]
[0179] Among them, c2 represents the capacity of the new cloud disk, h2 represents the predicted average IOPS, and g2 represents the predicted average bandwidth.
[0180] Step 8: Get the average of the three resource utilizations after the new cloud disk is added, and based on the average, get the Manhattan distance of the resource utilization. Use the storage node with the smallest Manhattan distance as the node for deploying the new cloud disk to complete the management of the cloud disk.
[0181] The Manhattan distance (vector (η' c1 ,η' h1 ,η' g1 ) and vector The Manhattan distance between
[0182]
[0183] in, Indicates the average utilization of the three resources after the new cloud disk is added.
[0184] Through the above solution, the optimal allocation of new cloud disks can be achieved, load balancing can be ensured, the resource utilization of storage nodes and the number of cloud disks that can be loaded can be effectively improved, and the deployment and management of cloud disks can be realized.
[0185] See also Figure 3 , Figure 3 This is a specific flow chart of a cloud disk allocation method provided in an embodiment of the present application. Figure 3 As shown, the method comprises the following steps:
[0186] Step 310: Obtain a training data set and an initial hyperparameter group; the training data set includes: historical cloud disk subscription information and historical cloud disk load information.
[0187] Step 315: clustering the cloud disk historical ordering information to obtain N clustering results.
[0188] Step 320: For the kth clustering result among the N clustering results, obtain target load information, wherein the target load information includes cloud disk historical load information corresponding to each cloud disk historical subscription information in the kth clustering result; and obtain the mean value of the target load information.
[0189] Step 325: Determine the mean of the target load information as the actual load information corresponding to all cloud disk historical ordering information in the kth clustering result; and use the actual load information as the true value label of all cloud disk historical ordering information in the kth clustering result.
[0190] Step 330: In the process of training the load prediction model to be trained based on the training data set, multiple loss values obtained by the load prediction model to be trained using multiple different hyperparameter groups are obtained; the multiple different hyperparameter groups are obtained based on multiple update values; the multiple update values include at least one update value among the first update value, the second update value, the third update value and the fourth update value.
[0191] In an embodiment of the present application, the first update value is obtained based on the first step length, the second update value is obtained based on the second step length, the third update value is obtained based on the third step length, the fourth update value is obtained based on the fourth step length, and the second step length is the same as the third step length; the multiple different hyperparameter groups include at least one hyperparameter group based on the first update value, the second hyperparameter group based on the second update value, the third hyperparameter group based on the third update value, and the fourth hyperparameter group based on the fourth update value.
[0192] The first update value can be determined by the following formula:
[0193] X i1 =X i +step1(2rand1-1);
[0194] Among them, X i1 is the first updated value, X i is the hyperparameter group currently used by the load prediction model to be trained, step1 is the first step length, and the first step length is obtained based on the current number of training times, and rand1 is used to represent a random number between (-1, 1).
[0195] The second update value may be determined in the following manner: for the ith initial hyperparameter group among the M initial hyperparameter groups, determining a first hyperparameter group corresponding to the ith initial hyperparameter group from the M initial hyperparameter groups, wherein the first hyperparameter group is a hyperparameter group whose Euclidean distance with the ith initial hyperparameter group is less than a set threshold; based on the first hyperparameter group, determining a central hyperparameter group corresponding to the ith initial hyperparameter group; obtaining a search factor and a second step length, wherein the search factor and the second step length are both determined according to the current number of training times; when the search factor and the central hyperparameter group satisfy a first condition, determining a second update value corresponding to the ith hyperparameter group based on the second step length and the central hyperparameter group by the following formula:
[0196]
[0197] Among them, X iis the i-th hyperparameter group, step2 is the second step length, rand2 is used to represent a random number between (0,1), X c is the central hyperparameter group, X i2 is the second updated value corresponding to the i-th hyperparameter group.
[0198] The third update value may be determined in the following manner: for the i-th initial hyperparameter group among the M initial hyperparameter groups, determine the first hyperparameter group corresponding to the i-th initial hyperparameter group from the M initial hyperparameter groups, wherein the first hyperparameter group is a hyperparameter group whose Euclidean distance with the i-th initial hyperparameter group is less than a set threshold; determine the hyperparameter group with the smallest loss value among the first hyperparameter groups of the i-th hyperparameter group as the second hyperparameter group corresponding to the i-th hyperparameter group; obtain a third step length, wherein the third step length is determined according to the current number of training times; and determine the third update value of the i-th hyperparameter group by the following formula:
[0199]
[0200] Among them, X i3 is the third update value of the i-th hyperparameter group, X i is the i-th hyperparameter group, rand2 is used to represent a random number between (0,1), step3 is used to represent the third step, and X max is the second hyperparameter group.
[0201] The fourth update value may be determined by: determining the target dimension parameter of the initial hyperparameter group; obtaining a fourth step length based on the current number of training times; and updating the target dimension parameter based on the fourth step length by the following formula to obtain an updated target dimension parameter: i4 =x id +step4*rand1; based on the updated target dimension parameter, obtain the fourth updated value; where x id is the target dimension parameter, x i4 is the updated target dimension parameter, step4 is the fourth step length, and rand1 is used to represent a random number between (-1, 1).
[0202] Step 335: Determine the hyperparameter group corresponding to the minimum loss value among the multiple loss values as the target hyperparameter group.
[0203] In an embodiment of the present application, after the hyperparameter group corresponding to the minimum loss value among the multiple loss values is determined as the target hyperparameter group, the method further includes: in the process of training the load prediction model to be trained, if the loss value of at least Q training times has not decreased, performing reverse learning and Cauchy mutation on the target hyperparameter group; wherein Q is a positive integer greater than 1.
[0204] Step 340: Based on the target hyperparameter group, obtain the target load prediction model.
[0205] Step 345: Obtain target ordering information of the target cloud disk, where the target cloud disk is the cloud disk applied for.
[0206] In the embodiment of the present application, the above steps 310 to 340 are a training process of the target load prediction model, and the training process may also be performed after step 345.
[0207] Step 350: Input the target ordering information into the target load prediction model to obtain the predicted load information corresponding to the target cloud disk; the target load prediction model is a deep learning model for load prediction, and the target hyperparameter group used by the target load prediction model is obtained based on the initial hyperparameter group and the target update value, and the target update value is used to update the initial hyperparameter group.
[0208] Step 355: Allocate the target cloud disk based on the predicted load information.
[0209] In an embodiment of the present application, the target subscription information of the target cloud disk is obtained, and the target cloud disk is the cloud disk applied for use; the target subscription information is input into the target load prediction model to obtain the predicted load information corresponding to the target cloud disk; the target load prediction model is a deep learning model for load prediction, and the target hyperparameter group used by the target load prediction model is obtained based on the initial hyperparameter group and the target update value, and the target update value is used to update the initial hyperparameter group; based on the predicted load information, the target cloud disk is allocated. In this way, the predicted load information of the target cloud disk is obtained through the target load prediction model, and the target cloud disk is further allocated through the predicted load information. In this process, the target hyperparameter group used by the cloud disk load prediction model is obtained based on the initial hyperparameter group and the target update value. By updating the initial hyperparameter group with the target update value, a more suitable target hyperparameter group can be selected, so that the target load prediction model has a better load prediction effect, so as to obtain a more suitable cloud disk allocation result, which solves the problem of poor cloud disk allocation results in the related technology.
[0210] It is important to understand that Figures 1 to 3 The explanations of the same or corresponding steps in the following can be referred to each other. Figure 1 The explanation of step 110 and step 120 in Figure 3 Step 345 and step 350 in .
[0211] At the same time, it should be understood that a cloud disk allocation method provided by an embodiment of the present application may have the following beneficial effects: First, the present application first clusters the historical ordering information of the cloud disk, and averages the load information based on the clustering results to obtain a training data pair for training the model, and realizes the load prediction of the new cloud disk through the trained target load prediction model, and finally deploys the new cloud disk according to the load information obtained by the load prediction, thereby realizing the load balancing of the storage node. Compared with the direct deployment of related technologies, it can realize the optimal allocation of the new cloud disk, thereby effectively improving the resource utilization of the storage node and the number of cloud disks that can be loaded, and realizing the optimized management of the cloud disk. Second, the present application proposes an optimization method for the hyperparameter group of the target load prediction model, which can realize the training of the load prediction model relatively quickly, and will not fall into the local optimum, and can also realize local refined search, thereby realizing the acquisition of the optimal hyperparameter group, so as to ensure that the target load prediction model has a better load prediction effect.
[0212] See also Figure 4 , Figure 4 is a structural block diagram of a cloud disk allocation device provided in an embodiment of the present application. Figure 4 As shown, an embodiment of the present application provides a cloud disk allocation device 400 , and the cloud disk allocation device 400 includes: an acquisition module 410 , a prediction module 420 and an allocation module 430 .
[0213] The acquisition module 410 is used to acquire target subscription information of a target cloud disk, where the target cloud disk is the cloud disk applied for;
[0214] The prediction module 420 is used to input the target ordering information into a target load prediction model to obtain predicted load information corresponding to the target cloud disk; the target load prediction model is a deep learning model for load prediction, and the target hyperparameter group used by the target load prediction model is obtained based on the initial hyperparameter group and the target update value, and the target update value is used to update the initial hyperparameter group;
[0215] The allocation module 430 is used to allocate the target cloud disk based on the predicted load information.
[0216] In an embodiment of the present application, the target subscription information of the target cloud disk is obtained, and the target cloud disk is the cloud disk applied for use; the target subscription information is input into the target load prediction model to obtain the predicted load information corresponding to the target cloud disk; the target load prediction model is a deep learning model for load prediction, and the target hyperparameter group used by the target load prediction model is obtained based on the initial hyperparameter group and the target update value, and the target update value is used to update the initial hyperparameter group; based on the predicted load information, the target cloud disk is allocated. In this way, the predicted load information of the target cloud disk is obtained through the target load prediction model, and the target cloud disk is further allocated through the predicted load information. In this process, the target hyperparameter group used by the cloud disk load prediction model is obtained based on the initial hyperparameter group and the target update value. By updating the initial hyperparameter group with the target update value, a more suitable target hyperparameter group can be selected, so that the target load prediction model has a better load prediction effect, so as to obtain a more suitable cloud disk allocation result, which solves the problem of poor cloud disk allocation results in the related technology.
[0217] The cloud disk allocation device provided in the embodiment of the present application can implement each process implemented in the above method embodiment. To avoid repetition, it will not be repeated here.
[0218] like Figure 5 As shown, an embodiment of the present application also provides an electronic device 500. The electronic device 500 includes: a processor 510 and a memory 520, and the memory 520 stores programs or instructions, and the programs or instructions are executed by the processor 510 to implement the steps of any one of the methods described above. For example, when the program is executed by the processor 510, the following process is implemented: obtaining target subscription information of the target cloud disk, the target cloud disk is the cloud disk applied for use; inputting the target subscription information into the target load prediction model to obtain the predicted load information corresponding to the target cloud disk; the target load prediction model is a deep learning model for load prediction, and the target hyperparameter group used by the target load prediction model is obtained based on the initial hyperparameter group and the target update value, and the target update value is used to update the initial hyperparameter group; based on the predicted load information, the target cloud disk is allocated. In this way, the predicted load information of the target cloud disk is obtained through the target load prediction model, and the target cloud disk is further allocated according to the predicted load information. In this process, the target hyperparameter group used by the cloud disk load prediction model is obtained based on the initial hyperparameter group and the target update value. By updating the initial hyperparameter group with the target update value, a more suitable target hyperparameter group can be selected, so that the target load prediction model has a better load prediction effect, so as to obtain a more suitable cloud disk allocation result, thereby solving the problem of poor cloud disk allocation results in related technologies.
[0219] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of each embodiment of the cloud disk allocation method are implemented and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.
[0220] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.
[0221] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0222] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned method embodiment and can achieve the same technical effect. To avoid repetition, it will not be repeated here.
[0223] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.
[0224] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.
[0225] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.
Claims
1. A cloud disk allocation method, characterized in that: include: Obtain target subscription information of a target cloud disk, where the target cloud disk is the cloud disk applied for; The target ordering information is input into a target load prediction model to obtain predicted load information corresponding to the target cloud disk; the target load prediction model is a deep learning model for load prediction, and the target hyperparameter group used by the target load prediction model is obtained based on an initial hyperparameter group and a target update value, and the target update value is used to update the initial hyperparameter group; Based on the predicted load information, the target cloud disk is allocated.
2. The method according to claim 1, characterized in that The training process of the target load prediction model includes: Get the training dataset and initial hyperparameter set; In the process of training the load prediction model to be trained based on the training data set, a plurality of loss values obtained by the load prediction model to be trained using a plurality of different hyperparameter groups are obtained; the plurality of different hyperparameter groups are obtained based on a plurality of update values; Determine the hyperparameter group corresponding to the minimum loss value among the multiple loss values as the target hyperparameter group; Based on the target hyperparameter group, obtaining the target load prediction model; The load prediction model to be trained initially uses the initial hyperparameter group, and the target load prediction model uses the target hyperparameter group.
3. The method according to claim 2, characterized in that The multiple update values include at least one update value of a first update value, a second update value, a third update value, and a fourth update value; the first update value is obtained based on the first step length, the second update value is obtained based on the second step length, the third update value is obtained based on the third step length, the fourth update value is obtained based on the fourth step length, and the second step length is the same as the third step length; The multiple different hyperparameter groups include at least one hyperparameter group based on the first update value, the second hyperparameter group based on the second update value, the third hyperparameter group based on the third update value, and the fourth hyperparameter group based on the fourth update value.
4. The method according to claim 2, characterized in that: The training data set includes: historical cloud disk ordering information and historical cloud disk load information; before training the load prediction model to be trained, the method further includes: Performing clustering processing on the cloud disk historical ordering information to obtain N clustering results, where N is a positive integer; For a k-th clustering result among the N clustering results, obtaining target load information, the target load information including cloud disk historical load information corresponding to each cloud disk historical subscription information in the k-th clustering result; obtaining a mean value of the target load information, where k is a positive integer less than or equal to N; The mean value of the target load information is determined as the actual load information corresponding to all the historical cloud disk ordering information in the kth clustering result; and the actual load information is used as the true value label of all the historical cloud disk ordering information in the kth clustering result.
5. The method according to claim 2, characterized in that: After determining the hyperparameter group corresponding to the minimum loss value among the multiple loss values as the target hyperparameter group, the method further includes: In the process of training the load prediction model to be trained, if the loss value of at least Q training times does not decrease, performing reverse learning and Cauchy mutation on the target hyperparameter group; Wherein, Q is a positive integer greater than 1.
6. The method according to claim 2 or 3, characterized in that: The multiple update values include a first update value, and the target update value is obtained based on the first update value; based on the first step length and the hyperparameter group currently used by the load prediction model to be trained, the first update value is determined in the following manner: X i1 =X i +step1(2rand1-1); Among them, X i1 is the first updated value, X i is the hyperparameter group currently used by the load prediction model to be trained, step1 is the first step length, and the first step length is obtained based on the current number of training times, and rand1 is used to represent a random number between (-1, 1).
7. The method according to claim 2 or 3, characterized in that: The initial hyperparameter group includes M initial hyperparameter groups in vector form, the multiple update values include a second update value, and the target update value is obtained based on the second update value; the second update value is determined in the following manner: For an i-th initial hyperparameter group among the M initial hyperparameter groups, determining a first hyperparameter group corresponding to the i-th initial hyperparameter group from the M initial hyperparameter groups, wherein the first hyperparameter group is a hyperparameter group whose Euclidean distance with the i-th initial hyperparameter group is less than a set threshold; Based on the first hyperparameter group, determining a central hyperparameter group corresponding to the i-th initial hyperparameter group; Obtaining a search factor and a second step length, wherein the search factor and the second step length are both determined according to the current number of training times; When the search factor and the central hyperparameter group satisfy a first condition, a second update value corresponding to the i-th hyperparameter group is determined based on the second step size and the central hyperparameter group.
8. The method according to claim 7, characterized in that Determining a second update value corresponding to the i-th hyperparameter group based on the second step size and the central hyperparameter group includes: Based on the second step size and the central hyperparameter group, the second update value corresponding to the i-th hyperparameter group is determined by the following formula: Among them, X i is the i-th hyperparameter group, step2 is the second step length, rand2 is used to represent a random number between (0,1), X c is the central hyperparameter group, X i2 is the second updated value corresponding to the i-th hyperparameter group.
9. The method according to claim 2 or 3, characterized in that: The initial hyperparameter group includes M initial hyperparameter groups in vector form, the multiple update values include a third update value, and the target update value is obtained based on the third update value; the third update value is determined in the following manner: For an i-th initial hyperparameter group among the M initial hyperparameter groups, determining a first hyperparameter group corresponding to the i-th initial hyperparameter group from the M initial hyperparameter groups, wherein the first hyperparameter group is a hyperparameter group whose Euclidean distance with the i-th initial hyperparameter group is less than a set threshold; Determine the hyperparameter group with the smallest loss value in the first hyperparameter group corresponding to the i-th hyperparameter group as the second hyperparameter group corresponding to the i-th hyperparameter group; Obtaining a third step length, wherein the third step length is determined according to the current number of training times; Based on the third step size and the second hyperparameter group, a third updated value of the i-th hyperparameter group is determined by the following formula: Among them, X i3 is the third update value corresponding to the i-th hyperparameter group, X i is the i-th hyperparameter group, rand2 is used to represent a random number between (0,1), step3 is used to represent the third step length, and X max is the second hyperparameter group.
10. The method according to claim 2 or 3, characterized in that: The multiple update values include a fourth update value, and the target update value includes a value obtained based on the fourth update value; the fourth update value is determined by: Determining target dimension parameters of the initial hyperparameter group; Based on the current number of training times, obtain the fourth step length; Based on the fourth step length, the target dimension parameter is updated by the following formula to obtain an updated target dimension parameter: x i4 =x id +step4*rand1; Based on the updated target dimension parameter, obtaining a fourth updated value; Among them, x id is the target dimension parameter, x i4 is the updated target dimension parameter, step4 is the fourth step length, and rand1 is used to represent a random number between (-1, 1).
