Concurrent instance management method and device, equipment and storage medium

Abstract

The invention relates to a concurrent instance management method and device, equipment and a storage medium. The method comprises the steps of obtaining storage bandwidth upper limit data in response to a distributed computing task submitted by a user, determining the number of concurrent instances based on the storage bandwidth upper limit data and a pre-trained prediction model, creating the concurrent instances based on the number of the concurrent instances, and processing the distributed computing task through the concurrent instances. Compared with the prior art, according to the embodiment of the invention, the number of the concurrent instances is determined on the basis of the storage bandwidth upper limit data and the pre-trained prediction model, and the concurrent instances are created on the basis of the number of the concurrent instances, so that the distributed computing task is processed through the concurrent instances. The number of the concurrent instances can be accurately predicted according to the upper limit data of the storage bandwidth, the number of the concurrent instances can be adjusted without manual intervention of a user, the number of trial and error times of the user is reduced, and time and computing resources of the user can be saved.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to a concurrent instance management method, apparatus, device and storage medium. Background Art

[0002] Currently, users can use a fully managed one-stop large-scale data processing and analysis platform (such as KMRServerless Spark) to perform large-scale distributed parallel computing, thereby reducing the time required for batch computing. Usually, the computing-storage architecture provided by the platform for users is storage-computing separation, that is, data and computing separation. The computing unit needs to pull data from the storage before the calculation to complete the calculation. After the calculation is completed, the computing unit needs to write to the storage concurrently. This large-scale access to storage requires a large bandwidth for storage. Considering that the storage is based on a multi-tenant architecture, the flow of individual users will be limited.

[0003] In the prior art, when a user submits a batch computing task, a large number of concurrent instances will be created in the platform, which will generate a large amount of storage bandwidth requirements. However, due to the storage side's policy of limiting the flow of a single user, when the bandwidth demand exceeds the upper limit of the storage bandwidth limit, the computing task will fail in part, causing the batch task to fail. When the user finds that the task has failed, he needs to intervene manually to reduce the concurrent instances of the computing task, thereby reducing the computing concurrency and then reducing the bandwidth generated by the concurrency.

[0004] However, the above method requires the user to manually intervene to adjust the number of concurrent instances, and the user cannot accurately set a reasonable number of concurrent instances at one time, and can only perform trial and error based on experience, which may waste a lot of user time. Summary of the invention

[0005] In order to solve the above technical problems, the present disclosure provides a concurrent instance management method, device, equipment and storage medium, which do not require manual intervention, reduce the number of user trial and error, and save user time and computing resources.

[0006] In a first aspect, an embodiment of the present disclosure provides a concurrent instance management method, the method comprising:

[0007] In response to a user submitting a distributed computing task, obtaining storage bandwidth upper limit data;

[0008] Determine the number of concurrent instances based on the storage bandwidth upper limit data and the pre-trained prediction model;

[0009] A concurrent instance is created based on the number of the concurrent instances to process the distributed computing task through the concurrent instance.

[0010] In some embodiments, determining the number of concurrent instances based on the storage bandwidth upper limit data and a pre-trained prediction model includes:

[0011] The storage bandwidth upper limit data is input into a pre-trained prediction model, and the number of instances is predicted by the pre-trained prediction model to obtain the number of concurrent instances.

[0012] In some embodiments, before determining the number of concurrent instances based on the storage bandwidth upper limit data and the pre-trained prediction model, the method further includes:

[0013] Create an initial forecast model;

[0014] The initial prediction model is trained to obtain a pre-trained prediction model.

[0015] In some embodiments, the creating an initial prediction model includes:

[0016] Constructing a linear regression equation, wherein the linear regression equation is used to characterize the linear relationship between the input value and the output value;

[0017] An initial prediction model is created based on the linear regression equation.

[0018] In some embodiments, the training of the initial prediction model to obtain a pre-trained prediction model includes:

[0019] Acquire a training data set, where each set of data in the training data set includes storage bandwidth upper limit sample data and the corresponding actual number of concurrent instances;

[0020] Constructing a loss function based on the training data set, wherein the loss function is used to characterize the difference between the predicted number of concurrent instances and the actual number by the initial prediction model;

[0021] The model parameters of the initial prediction model are updated based on the loss function to obtain a pre-trained prediction model, wherein the model parameters include weight parameters and bias parameters.

[0022] In some embodiments, updating the model parameters of the initial prediction model based on the loss function to obtain a pre-trained prediction model includes:

[0023] Minimize the loss function and calculate the weight parameter and bias parameter corresponding to the minimum value of the loss function;

[0024] The model parameters of the initial prediction model are updated based on the weight parameters and bias parameters corresponding to when the loss function takes the minimum value to obtain a pre-trained prediction model.

[0025] In some embodiments, calculating the weight parameter and the bias parameter when the loss function takes the minimum value includes:

[0026] Calculating partial derivatives of weight parameters of the loss function, setting the partial derivatives of the weight parameters of the loss function to zero, and obtaining a first equation;

[0027] Calculating the partial derivative of the bias parameter of the loss function, setting the partial derivative of the bias parameter of the loss function to zero, and obtaining the second equation;

[0028] The first equation and the second equation are solved simultaneously to obtain the weight parameter and the bias parameter corresponding to the minimum value of the loss function.

[0029] In a second aspect, an embodiment of the present disclosure provides a concurrent instance management device, the device comprising:

[0030] An acquisition module, used for acquiring storage bandwidth upper limit data in response to a distributed computing task submitted by a user;

[0031] A determination module, configured to determine the number of concurrent instances based on the storage bandwidth upper limit data and a pre-trained prediction model;

[0032] A management module is used to create a concurrent instance based on the number of the concurrent instances, so as to process the distributed computing task through the concurrent instance.

[0033] In a third aspect, an embodiment of the present disclosure provides an electronic device, including:

[0034] Memory;

[0035] Processor; and

[0036] Computer programs;

[0037] The computer program is stored in the memory and is configured to be executed by the processor to implement the method as described in the first aspect.

[0038] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the method as described in the first aspect.

[0039] In a fifth aspect, an embodiment of the present disclosure further provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the method described in the first aspect is implemented.

[0040] The concurrent instance management method, apparatus, device and storage medium provided in the embodiments of the present disclosure obtain storage bandwidth upper limit data in response to a user submitting a distributed computing task, determine the number of concurrent instances based on the storage bandwidth upper limit data and a pre-trained prediction model, create a concurrent instance based on the number of concurrent instances, and process the distributed computing task through the concurrent instance. Compared with the prior art, the embodiments of the present disclosure determine the number of concurrent instances based on the storage bandwidth upper limit data and a pre-trained prediction model, create a concurrent instance based on the number of concurrent instances, and process the distributed computing task through the concurrent instance. The number of concurrent instances can be predicted more accurately based on the storage bandwidth upper limit data, without the need for the user to manually intervene to adjust the number of concurrent instances, thereby reducing the number of user trial and error times and saving user time and computing resources. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0043] Figure 1 A flow chart of a concurrent instance management method provided by an embodiment of the present disclosure;

[0044] Figure 2 A flowchart of a concurrent instance management method provided by another embodiment of the present disclosure;

[0045] Figure 3 A flowchart of a concurrent instance management method provided by another embodiment of the present disclosure;

[0046] Figure 4 A schematic diagram of the design principle of the prediction model provided in the embodiment of the present disclosure;

[0047] Figure 5 A schematic diagram of the structure of a concurrent instance management device provided in an embodiment of the present disclosure;

[0048] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0049] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features in the embodiments can be combined with each other without conflict.

[0050] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.

[0051] Currently, users can use a fully managed one-stop large-scale data processing and analysis platform (such as KMRServerless Spark) to perform large-scale distributed parallel computing, thereby reducing the time required for batch computing. Usually, the computing-storage architecture provided by the platform for users is storage-computing separation, that is, data and computing separation. The computing unit needs to pull data from the storage before the calculation to complete the calculation. After the calculation is completed, the computing unit needs to write to the storage concurrently. This large-scale access to storage requires a large amount of storage bandwidth. Considering that the storage is based on a multi-tenant architecture, the flow of individual users will be limited.

[0052] In the prior art, when a user submits a batch computing task, a large number of concurrent instances will be created in the platform, which will generate a large amount of storage bandwidth requirements. However, due to the storage side's policy of limiting the flow of a single user, when the bandwidth demand exceeds the upper limit of the storage bandwidth limit, the computing task will fail in part, causing the batch task to fail. When the user finds that the task has failed, he needs to intervene manually to reduce the concurrent instances of the computing task, thereby reducing the computing concurrency and then reducing the bandwidth generated by the concurrency.

[0053] However, the above method requires the user to manually intervene to adjust the number of concurrent instances, and the user cannot accurately set a reasonable number of concurrent instances at one time, and can only perform trial and error based on experience, which may waste a lot of user time.

[0054] To address this problem, an embodiment of the present disclosure provides a concurrent instance management method, which is described below in conjunction with a specific embodiment.

[0055] Figure 1A flowchart of a concurrent instance management method provided in an embodiment of the present disclosure. The execution subject of the method is an electronic device, which may be a portable mobile device such as a smart phone, a tablet computer, a laptop computer, or a fixed device such as a personal computer or a server, wherein the server may be a single server or a server cluster, and the server cluster may be a distributed cluster or a centralized cluster. The method may be applied to a scenario of managing concurrent instances, or to a scenario of determining the number of concurrent instances.

[0056] It is understandable that the concurrent instance management method provided by the embodiments of the present disclosure may also be applied in other scenarios.

[0057] Below Figure 1 The concurrent instance management method shown in the figure is introduced, and the method can be applied to electronic devices. The specific steps of the method are as follows:

[0058] S101. In response to a user submitting a distributed computing task, obtaining storage bandwidth upper limit data.

[0059] In this step, the user submits a distributed computing task, i.e., a batch computing task, and the electronic device obtains the storage bandwidth upper limit data in response to the user submitting the distributed computing task. Specifically, the electronic device reads the storage bandwidth upper limit data of the user. Optionally, the storage bandwidth upper limit data represents the upper limit value of the storage bandwidth allocated to the user, for example, the storage bandwidth upper limit data of the user is 300Gb / s, without limitation.

[0060] S102: Determine the number of concurrent instances based on the storage bandwidth upper limit data and the pre-trained prediction model.

[0061] In this step, a prediction model is pre-trained in the electronic device to predict the number of concurrent instances. After obtaining the storage bandwidth upper limit data, the electronic device will determine the number of concurrent instances based on the storage bandwidth upper limit data and the pre-trained prediction model. For example, the storage bandwidth upper limit that the user can use is 500Gb / s. Based on the storage bandwidth upper limit data and the pre-trained prediction model, it is determined that the maximum number of concurrent instances that can be submitted for this computing task is 10,000 concurrent instances. When the number of concurrent instances exceeds 10,000, the bandwidth required for this batch processing exceeds the storage bandwidth upper limit, causing the batch processing task to fail.

[0062] In some embodiments, determining the number of concurrent instances based on the storage bandwidth upper limit data and a pre-trained prediction model includes: inputting the storage bandwidth upper limit data into the pre-trained prediction model, predicting the number of instances through the pre-trained prediction model, and obtaining the number of concurrent instances.

[0063] In this embodiment, after obtaining the storage bandwidth upper limit data, the electronic device can input the storage bandwidth upper limit data into a pre-trained prediction model, and predict the number of instances through the pre-trained prediction model to obtain the number of concurrent instances. The number of concurrent instances can be predicted more accurately based on the storage bandwidth upper limit data, without the need for manual intervention by the user to adjust the number of concurrent instances, thereby reducing the number of user trial and error times and saving user time and computing resources.

[0064] S103: Create a concurrent instance based on the number of the concurrent instances to process the distributed computing task through the concurrent instance.

[0065] In this step, the electronic device creates a concurrent instance according to the number of concurrent instances, and further processes the distributed computing task through the concurrent instance to obtain a processing result. For example, if the number of concurrent instances is 100, the electronic device creates 100 concurrent instances.

[0066] The disclosed embodiment obtains storage bandwidth upper limit data in response to a user submitting a distributed computing task, determines the number of concurrent instances based on the storage bandwidth upper limit data and a pre-trained prediction model, creates a concurrent instance based on the number of concurrent instances, and processes the distributed computing task through the concurrent instance. Compared with the prior art, the disclosed embodiment determines the number of concurrent instances based on the storage bandwidth upper limit data and a pre-trained prediction model, creates a concurrent instance based on the number of concurrent instances, and processes the distributed computing task through the concurrent instance. The number of concurrent instances can be predicted more accurately based on the storage bandwidth upper limit data, without the need for the user to manually intervene to adjust the number of concurrent instances, thereby reducing the number of user trial and error times and saving user time and computing resources.

[0067] Figure 2 A flowchart of a concurrent instance management method provided by another embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the method includes the following steps:

[0068] S201. In response to a user submitting a distributed computing task, obtaining storage bandwidth upper limit data.

[0069] Specifically, the implementation process and principle of S201 and S101 are the same, and will not be repeated here.

[0070] S202: Create an initial prediction model.

[0071] In this step, the user will perform an operation to create a model, and the electronic device will create an initial prediction model in response to the user's operation to create a model.

[0072] In some embodiments, S202 may include but is not limited to S2021 and S2022:

[0073] S2021. Construct a linear regression equation, where the linear regression equation is used to characterize the linear relationship between the input value and the output value.

[0074] According to data sampling and observation, the storage bandwidth limit and the number of concurrent instances are as follows: Figure 4 The linear relationship shown, where the y-axis represents the number of concurrent instances and the x-axis represents the upper limit of the storage bandwidth, so a linear regression equation is constructed to fit the relationship between the independent variable (input value) and the dependent variable (output value). Optionally, the linear regression equation is expressed as: y=wx+b, y represents the number of concurrent instances, x represents the upper limit of the storage bandwidth, w is the weight (slope), which represents the degree of influence of the feature on the target, and b is the bias (intercept), which represents the intersection of the linear function and the vertical axis. The linear regression equation is used to characterize the linear relationship between the input value and the output value, that is, the linear relationship between the upper limit of the storage bandwidth and the number of concurrent instances.

[0075] S2022. Create an initial prediction model based on the linear regression equation.

[0076] Furthermore, an initial prediction model was created based on the linear regression equation.

[0077] S203: Train the initial prediction model to obtain a pre-trained prediction model.

[0078] In this step, the electronic device may train the initial prediction model to obtain a pre-trained prediction model.

[0079] In some embodiments, S203 may include but is not limited to S2031, S2032, S2033:

[0080] S2031, obtaining a training data set, wherein each set of data in the training data set includes storage bandwidth upper limit sample data and the corresponding actual number of concurrent instances;

[0081] For example, using x i Indicates the upper limit of the i-th storage bandwidth, y i represents the number of concurrent instances of the i-th instance. The training dataset can be expressed as: {(x 1 ,y 1 ), (x 2 ,y 2 ),...,(x n ,y n )}.

[0082] S2032, constructing a loss function based on the training data set, where the loss function is used to characterize the difference between the predicted number of concurrent instances and the actual number by the initial prediction model;

[0083] In order to find the optimal w and b, we need to construct a loss function to measure the difference between the model's predicted quantity and the actual quantity. The commonly used loss function is the mean square error (MSE), which is expressed as follows:

[0084]

[0085] S2033. Update the model parameters of the initial prediction model based on the loss function to obtain a pre-trained prediction model, wherein the model parameters include weight parameters and bias parameters.

[0086] In this step, the model parameters of the initial prediction model can be updated according to the loss function to obtain the weight parameters of the pre-trained prediction model. w is the weight parameter and b is the bias parameter.

[0087] S204: Determine the number of concurrent instances based on the storage bandwidth upper limit data and the pre-trained prediction model.

[0088] Specifically, the implementation process and principle of S204 and S102 are the same, and will not be repeated here.

[0089] S205. Create a concurrent instance based on the number of the concurrent instances to process the distributed computing task through the concurrent instance.

[0090] Specifically, the implementation process and principle of S205 and S103 are the same, and will not be repeated here.

[0091] The embodiment of the present disclosure obtains storage bandwidth upper limit data and creates an initial prediction model in response to a user submitting a distributed computing task. Further, the initial prediction model is trained to obtain a pre-trained prediction model, and the number of concurrent instances is determined based on the storage bandwidth upper limit data and the pre-trained prediction model. Then, a concurrent instance is created based on the number of concurrent instances to process the distributed computing task through the concurrent instance. Compared with the prior art, the embodiment of the present disclosure determines the number of concurrent instances based on the storage bandwidth upper limit data and the pre-trained prediction model, creates a concurrent instance based on the number of concurrent instances, and processes the distributed computing task through the concurrent instance. The number of concurrent instances can be predicted more accurately based on the storage bandwidth upper limit data, without the need for the user to manually intervene to adjust the number of concurrent instances, thereby reducing the number of user trial and error times and saving user time and computing resources.

[0092] Figure 3 A flowchart of a concurrent instance management method provided by another embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the method includes the following steps:

[0093] S301. In response to a user submitting a distributed computing task, obtaining storage bandwidth upper limit data.

[0094] Specifically, the implementation process and principle of S301 and S101 are the same, and will not be repeated here.

[0095] S302: Create an initial prediction model.

[0096] Specifically, the implementation process and principle of S302 and S202 are the same, and will not be repeated here.

[0097] S303: Acquire a training data set, where each set of data in the training data set includes storage bandwidth upper limit sample data and the corresponding actual number of concurrent instances.

[0098] Specifically, the implementation process and principle of S303 and S2031 are the same, and will not be repeated here.

[0099] S304: construct a loss function based on the training data set, where the loss function is used to characterize the difference between the predicted number of concurrent instances and the actual number by the initial prediction model.

[0100] Specifically, the implementation process and principle of S304 and S2032 are the same, and will not be repeated here.

[0101] S305: Minimize the loss function and calculate the weight parameter and bias parameter corresponding to the minimum value of the loss function.

[0102] In order to obtain the optimal w and b, the loss function J(w,b) needs to be minimized. In this step, the electronic device minimizes the loss function and calculates the weight parameters and bias parameters corresponding to the minimum value of the loss function. Optionally, the minimum value can be achieved by solving the partial derivative of the loss function and setting it to zero.

[0103] In some embodiments, the weight parameters and bias parameters corresponding to the minimum value of the loss function calculated in S305 may include but are not limited to S3051, S3052, and S3053:

[0104] S3051, calculating the partial derivative of the weight parameter of the loss function, setting the partial derivative of the weight parameter of the loss function to zero, and obtaining a first equation;

[0105] In this step, the partial derivative of the weight parameter w of the loss function J(w,b) is calculated:

[0106]

[0107] Let the partial derivative of the weight parameter of the loss function be zero, that is The first equation is obtained:

[0108]

[0109] S3052, calculating a partial derivative of the bias parameter of the loss function, setting the partial derivative of the bias parameter of the loss function to zero, and obtaining a second equation;

[0110] In this step, the partial derivative of the bias parameter b of the loss function J(w,b) is calculated:

[0111]

[0112] Let the partial derivative of the bias parameter of the loss function be zero, that is The second equation is obtained:

[0113]

[0114] S3053. Solve the first equation and the second equation simultaneously to obtain the weight parameter and the bias parameter corresponding to the minimum value of the loss function.

[0115] In this step, the first equation and the second equation are solved simultaneously, and the second equation can be simplified as follows:

[0116]

[0117] Substituting b into the first equation, we obtain:

[0118]

[0119]

[0120] By using the above formula, w can be obtained, and b can be further obtained to obtain the weight parameter and bias parameter corresponding to the minimum value of the loss function.

[0121] S306: Update the model parameters of the initial prediction model based on the weight parameters and bias parameters corresponding to when the loss function takes a minimum value, to obtain a pre-trained prediction model.

[0122] In this step, after obtaining the weight parameters and bias parameters corresponding to when the loss function takes the minimum value, the electronic device updates the model parameters of the initial prediction model according to the weight parameters and bias parameters corresponding to when the loss function takes the minimum value, and obtains the pre-trained prediction model. Specifically, the model parameters of the initial prediction model are updated to the weight parameters and bias parameters corresponding to when the loss function takes the minimum value, the model training is completed, and the pre-trained prediction model is obtained.

[0123] S307: Determine the number of concurrent instances based on the storage bandwidth upper limit data and the pre-trained prediction model.

[0124] Specifically, the implementation process and principle of S307 and S102 are the same, and will not be repeated here.

[0125] S308. Create a concurrent instance based on the number of the concurrent instances to process the distributed computing task through the concurrent instance.

[0126] Specifically, the implementation process and principle of S308 and S103 are the same, and will not be repeated here.

[0127] The disclosed embodiment obtains storage bandwidth upper limit data, creates an initial prediction model, and obtains a training data set in response to a user submitting a distributed computing task. Each set of data in the training data set includes storage bandwidth upper limit sample data and the corresponding actual number of concurrent instances. Further, a loss function is constructed based on the training data set, and the loss function is used to characterize the difference between the predicted number and the actual number of concurrent instances by the initial prediction model. The loss function is minimized, and the weight parameters and bias parameters corresponding to the minimum value of the loss function are calculated. The model parameters of the initial prediction model are updated based on the weight parameters and bias parameters corresponding to the minimum value of the loss function to obtain a pre-trained prediction model. Then, the number of concurrent instances is determined based on the storage bandwidth upper limit data and the pre-trained prediction model, and a concurrent instance is created based on the number of concurrent instances to process the distributed computing task through the concurrent instance. Through this method, the number of concurrent instances can be more accurately predicted based on the storage bandwidth upper limit data, without the need for the user to manually intervene to adjust the number of concurrent instances, reducing the number of user trial and error times, and saving user time and computing resources.

[0128] Figure 5 Schematic diagram of the structure of the concurrent instance management device provided in the embodiment of the present disclosure. The concurrent instance management device may be the electronic device as described in the above embodiment, or the concurrent instance management device may be a component or assembly in the electronic device. The concurrent instance management device provided in the embodiment of the present disclosure may execute the processing flow provided in the concurrent instance management method embodiment, such as Figure 5 As shown, the concurrent instance management device 50 includes: an acquisition module 51, a determination module 52, and a management module 53; wherein the acquisition module 51 is used to obtain storage bandwidth upper limit data in response to a user submitting a distributed computing task; the determination module 52 is used to determine the number of concurrent instances based on the storage bandwidth upper limit data and a pre-trained prediction model; the management module 53 is used to create concurrent instances based on the number of concurrent instances to process the distributed computing task through the concurrent instances.

[0129] Optionally, when the determination module 52 determines the number of concurrent instances based on the storage bandwidth upper limit data and the pre-trained prediction model, it is specifically used to: input the storage bandwidth upper limit data into the pre-trained prediction model, predict the number of instances through the pre-trained prediction model, and obtain the number of concurrent instances.

[0130] Optionally, before determining the number of concurrent instances based on the storage bandwidth upper limit data and the pre-trained prediction model, the device 50 also includes: a creation module 54 and / or an acquisition module 55; the creation module 54 is used to create an initial prediction model; the acquisition module 55 is used to train the initial prediction model to obtain a pre-trained prediction model.

[0131] Optionally, when the creation module 54 creates the initial prediction model, it is specifically used to: construct a linear regression equation, where the linear regression equation is used to characterize the linear relationship between the input value and the output value; and create the initial prediction model based on the linear regression equation.

[0132] Optionally, when the obtaining module 55 trains the initial prediction model to obtain a pre-trained prediction model, it is specifically used to: obtain a training data set, each group of data in the training data set includes storage bandwidth upper limit sample data and the corresponding actual number of concurrent instances; construct a loss function based on the training data set, and the loss function is used to characterize the difference between the predicted number and the actual number of concurrent instances by the initial prediction model; update the model parameters of the initial prediction model based on the loss function to obtain a pre-trained prediction model, and the model parameters include weight parameters and bias parameters.

[0133] Optionally, the obtaining module 55 updates the model parameters of the initial prediction model based on the loss function to obtain a pre-trained prediction model, and is specifically used to: minimize the loss function, calculate the weight parameters and bias parameters corresponding to the minimum value of the loss function; update the model parameters of the initial prediction model based on the weight parameters and bias parameters corresponding to the minimum value of the loss function, and obtain the pre-trained prediction model.

[0134] Optionally, when the obtaining module 55 calculates the weight parameters and bias parameters when the loss function takes the minimum value, it is specifically used to: find the partial derivative of the weight parameters of the loss function, set the partial derivative of the weight parameters of the loss function to zero, and obtain the first equation; find the partial derivative of the bias parameters of the loss function, set the partial derivative of the bias parameters of the loss function to zero, and obtain the second equation; solve the first equation and the second equation jointly to obtain the weight parameters and bias parameters corresponding to the minimum value of the loss function.

[0135] Figure 5The concurrent instance management device of the illustrated embodiment can be used to execute the technical solution of the above-mentioned method embodiment. Its implementation principle and technical effect are similar and will not be repeated here.

[0136] Figure 6 Schematic diagram of the structure of an electronic device in the embodiment of the present disclosure. Figure 6 , which shows a structural schematic diagram of an electronic device 600 suitable for implementing the embodiments of the present disclosure. Figure 6 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0137] like Figure 6 As shown, the electronic device 600 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage device 608 into a random access memory (RAM) 603 to implement the concurrent instance management method of the embodiment described in the present disclosure. In the RAM 603, various programs and data required for the operation of the electronic device 600 are also stored. The processing device 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0138] Typically, the following devices may be connected to the I / O interface 605: an input device 606 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 607 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 608 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 609. The communication device 609 may allow the electronic device 600 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 6 The electronic device 600 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0139] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a non-transitory computer-readable medium, and the computer program contains a program code for executing the method shown in the flowchart, thereby implementing the concurrent instance management method as described above. In such an embodiment, the computer program can be downloaded and installed from the network through the communication device 609, or installed from the storage device 608, or installed from the ROM 602. When the computer program is executed by the processing device 601, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0140] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0141] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0142] The computer-readable medium may be included in the electronic device, or may exist independently without being installed in the electronic device.

[0143] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device:

[0144] In response to a user submitting a distributed computing task, obtaining storage bandwidth upper limit data;

[0145] Determine the number of concurrent instances based on the storage bandwidth upper limit data and the pre-trained prediction model;

[0146] A concurrent instance is created based on the number of the concurrent instances to process the distributed computing task through the concurrent instance.

[0147] Optionally, when the above one or more programs are executed by the electronic device, the electronic device may also execute other steps described in the above embodiments.

[0148] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including, but not limited to, object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0149] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0150] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit does not, in some cases, limit the unit itself.

[0151] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0152] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0153] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0154] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0155] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.

Claims

1. A concurrent instance management method, characterized in that: The method comprises: In response to a user submitting a distributed computing task, obtaining storage bandwidth upper limit data; Determine the number of concurrent instances based on the storage bandwidth upper limit data and the pre-trained prediction model; A concurrent instance is created based on the number of the concurrent instances to process the distributed computing task through the concurrent instance.

2. The method according to claim 1, characterized in that The determining the number of concurrent instances based on the storage bandwidth upper limit data and the pre-trained prediction model includes: The storage bandwidth upper limit data is input into a pre-trained prediction model, and the number of instances is predicted by the pre-trained prediction model to obtain the number of concurrent instances.

3. The method according to claim 1, characterized in that Before determining the number of concurrent instances based on the storage bandwidth upper limit data and the pre-trained prediction model, the method further includes: Create an initial forecast model; The initial prediction model is trained to obtain a pre-trained prediction model.

4. The method according to claim 3, characterized in that The step of creating an initial prediction model comprises: Constructing a linear regression equation, wherein the linear regression equation is used to characterize the linear relationship between the input value and the output value; An initial prediction model is created based on the linear regression equation.

5. The method according to claim 3, characterized in that: The training of the initial prediction model to obtain a pre-trained prediction model includes: Acquire a training data set, where each set of data in the training data set includes storage bandwidth upper limit sample data and the corresponding actual number of concurrent instances; Constructing a loss function based on the training data set, wherein the loss function is used to characterize the difference between the predicted number of concurrent instances and the actual number by the initial prediction model; The model parameters of the initial prediction model are updated based on the loss function to obtain a pre-trained prediction model, wherein the model parameters include weight parameters and bias parameters.

6. The method according to claim 5, characterized in that The updating of the model parameters of the initial prediction model based on the loss function to obtain the pre-trained prediction model includes: Minimize the loss function and calculate the weight parameter and bias parameter corresponding to the minimum value of the loss function; The model parameters of the initial prediction model are updated based on the weight parameters and bias parameters corresponding to when the loss function takes the minimum value to obtain a pre-trained prediction model.

7. The method according to claim 6, characterized in that The calculating of the weight parameter and the bias parameter when the loss function takes the minimum value includes: Calculating partial derivatives of weight parameters of the loss function, setting the partial derivatives of the weight parameters of the loss function to zero, and obtaining a first equation; Calculating the partial derivative of the bias parameter of the loss function, setting the partial derivative of the bias parameter of the loss function to zero, and obtaining the second equation; The first equation and the second equation are solved simultaneously to obtain the weight parameter and the bias parameter corresponding to the minimum value of the loss function.

8. A concurrent instance management device, characterized in that: include: An acquisition module, used for acquiring storage bandwidth upper limit data in response to a distributed computing task submitted by a user; A determination module, configured to determine the number of concurrent instances based on the storage bandwidth upper limit data and a pre-trained prediction model; A management module is used to create a concurrent instance based on the number of the concurrent instances, so as to process the distributed computing task through the concurrent instance.

9. An electronic device, characterized in that: include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and is configured to be executed by the processor to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images