Storage capacity calculation method and device, equipment, storage medium and program product

Abstract

The invention provides a storage capacity calculation method and device, equipment, a storage medium and a program product. The method comprises the steps of predicting a training data volume and an original data volume of a model for completing a single training task based on computing resource parameters of a first training cluster and training parameters required by the model deployed in the first training cluster for completing the single training task; and, based on the training parameters, predicting a model parameter data volume, a stage model state data volume, and a data volume of an optimizer for optimizing parameters of the model; calculating the task data volume of the model in the training process based on the training data volume, the original data volume, the model parameter data volume, the stage model state data volume and the data volume of the optimizer; and calculating the storage capacity required to be configured by the model training based on the task data volume. According to the scheme, scientific prediction of the storage capacity is realized, and the resource utilization rate can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a storage capacity calculation method, device, equipment, storage medium and program product. Background Art

[0002] As the scale of AI large-scale model training increases, the amount of training data also increases dramatically. The investment in storage resources has become the largest investment in the intelligent computing center in addition to intelligent computing power. It is not only related to training efficiency, but also related to the construction cost of the entire intelligent computing center. In multimodal training scenarios, the training data is more complex and diverse, and whether the ratio of intelligent computing power to storage resources is scientific and reasonable is more important.

[0003] However, the current intelligent computing center has no reasonable basis to follow in terms of computing and storage resources. Instead, it configures them based on experience. As a result, there is often waste of storage resources due to over-allocation, or insufficient storage resources for further expansion. Summary of the invention

[0004] In view of the above problems, the present application is proposed. The present application provides a storage capacity calculation method, device, equipment, storage medium and program product.

[0005] According to a first aspect of the present application, a storage capacity calculation method is provided, comprising:

[0006] Based on the computing resource parameters of the first training cluster and the training parameters required by the model deployed in the first training cluster to complete a single training task, predict the amount of training data and the amount of original data required by the model to complete the single training task; and based on the training parameters, predict the amount of model parameter data, the amount of staged model state data, and the amount of data of the optimizer used to optimize the parameters of the model;

[0007] Calculate the task data volume of the model during the training process based on the training data volume, the original data volume, the model parameter data volume, the staged model state data volume and the optimizer data volume;

[0008] Based on the amount of task data, calculate the storage capacity required for model training.

[0009] In addition, according to the storage capacity calculation method of the first aspect of the present application, after calculating the storage capacity required for the model training based on the task data volume, it also includes:

[0010] Calculating a computing-storage ratio based on the storage capacity and the computing resource parameters;

[0011] The computing-storage ratio and the computing resource parameters of the second training cluster are used to calculate the storage capacity required for model training deployed in the second training cluster.

[0012] In addition, according to the storage capacity calculation method of the first aspect of the present application, based on the computing resource parameters of the first training cluster, the amount of training data and the amount of original data of the model deployed in the training cluster to complete a single training task are predicted; and based on the model parameter amount of the model, the amount of model parameter data, the amount of staged model state data, and the amount of data of the optimizer used to optimize the parameters of the model are predicted, and further comprising:

[0013] Calculating high-performance storage capacity based on the amount of training data, the amount of original data, the amount of staged model state data, and the amount of optimizer data; and

[0014] The capacity-based storage capacity is calculated based on the original data volume, the model parameter volume of the model, the staged model state data volume and the optimizer data volume.

[0015] In addition, according to the storage capacity calculation method of the first aspect of the present application, based on the computing resource parameters of the first training cluster and the training parameters required for the model deployed in the first training cluster to complete a single training task, predicting the amount of training data and the amount of original data required for the model to complete the single training task includes:

[0016] From the computing resource parameters, the number of processors configured in the first training cluster and the effective computing power of a single processor are obtained; and from the training parameters, the training duration of the single training task, the model parameter quantity of the model, the number of bytes occupied by the corpus, and the number of corpuses processed using computing power per unit parameter quantity are obtained;

[0017] Calculate the product of the effective computing power of the single processor, the number of the processors, and the training duration of the single training task to obtain a first operation result, where the first operation result is used to characterize the total number of floating-point operations required for the model to complete the single training task;

[0018] Taking the quotient of the first operation result and the model parameter quantity, a second operation result is obtained, where the second operation result is used to represent the number of floating-point operations corresponding to a single model parameter;

[0019] The product of the second operation result, the number of bytes occupied by the corpus and the number of corpora processed by the unit parameter amount using computing power is calculated to obtain the training data volume, and the quotient of the training data volume and a preset cleaning coefficient is calculated to obtain the original data volume.

[0020] In addition, according to the storage capacity calculation method of the first aspect of the present application, based on the training parameters, the amount of model parameter data, the amount of staged model state data, and the amount of data of the optimizer used to optimize the parameters of the model are predicted, including:

[0021] Obtaining the model parameter quantity and the number of bytes occupied by the model parameters of the model from the training parameters;

[0022] Calculate the product of the model parameter amount and the number of bytes occupied by the model parameter to obtain the model parameter data amount;

[0023] The product of the model parameter data volume and the preset stage-by-stage model state coefficient is calculated to obtain the data volume of the optimizer; and the model parameter data volume is used as the stage-by-stage model state data volume.

[0024] In addition, according to the storage capacity calculation method of the first aspect of the present application, based on the task data volume, calculating the storage capacity size required for the model training includes:

[0025] Calculating the product of the task data volume and the preset metadata ratio to obtain a third operation result;

[0026] The storage capacity is obtained by calculating a quotient of the third operation result and an upper limit ratio of storage devices of the first training cluster.

[0027] In addition, according to the storage capacity calculation method of the first aspect of the present application, based on the storage capacity size and the computing resource parameter, calculating the computing-storage ratio includes:

[0028] Based on the storage capacity, calculating the total number of floating-point operations of the first training cluster;

[0029] The calculation-to-memory ratio is obtained by calculating the quotient of the total number of floating-point operations, the number included in the computing resource parameter, and the effective computing power of a single processor.

[0030] According to a second aspect of the present application, a storage capacity calculation device is provided, comprising:

[0031] A prediction module, for predicting the amount of training data and the amount of original data required for the model to complete a single training task based on the computing resource parameters of the first training cluster and the training parameters required for the model deployed in the first training cluster to complete the single training task; and, based on the training parameters, predicting the amount of model parameter data, the amount of staged model state data, and the amount of data of the optimizer used to optimize the parameters of the model;

[0032] A first calculation module, used to calculate the task data volume of the model during the training process based on the training data volume, the original data volume, the model parameter data volume, the staged model state data volume and the optimizer data volume;

[0033] The second calculation module is used to calculate the storage capacity required for the model training based on the amount of task data.

[0034] According to a third aspect of the present application, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the method described in the first aspect.

[0035] According to a fourth aspect of the present application, a computer-readable storage medium is provided, on which a computer program / instruction is stored, and when the computer program / instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0036] According to a fifth aspect of the present application, there is provided a computer program product, comprising a computer program / instruction, characterized in that when the computer program / instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0037] As described in detail below, according to the storage capacity calculation method of the embodiment of the present application, the task-related data of artificial intelligence training is divided into multiple data categories, and modeling and calculation are performed separately to obtain the training data volume, the original data volume, the model parameter data volume, the staged model state data volume and the optimizer data volume, and the total task data volume of the model is calculated based on these data volumes, thereby calculating the storage capacity size that needs to be configured for the first training cluster during the model training process based on the total task data volume. It can be seen that the use of the solution of the present application realizes the scientific prediction of storage capacity, which helps to improve resource utilization.

[0038] It is to be understood that both the foregoing general description and the following detailed description are exemplary, and are intended to provide further explanation of the technology as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] By describing the embodiments of the present application in more detail in conjunction with the accompanying drawings, the above and other purposes, features and advantages of the present application will become more apparent. The accompanying drawings are used to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0040] Figure 1 It is a schematic diagram illustrating a large model training computing cluster and a storage cluster in the related technology.

[0041] Figure 2 is a flowchart illustrating the application of the storage capacity calculation method according to an embodiment of the present application.

[0042] Figure 3 is a schematic diagram illustrating an application of a storage capacity calculation device according to an embodiment of the present application.

[0043] Figure 4 It is a hardware block diagram of an electronic device according to an embodiment of the present application.

[0044] Figure 5 is a schematic diagram illustrating a computer-readable storage medium according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0046] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0047] The term "and / or" herein only describes an association relationship, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C may represent including any one or more elements selected from the set consisting of A, B, and C.

[0048] As the number of parameters in AI training models increases exponentially, the computing power of the models is growing exponentially. Training large models with large computing power inevitably involves access to a large amount of training data. Effective storage and efficient access to training data are the key to the efficiency of large model training, and are also important factors in whether intelligent computing power can fully exert its effectiveness.

[0049] In the current construction of intelligent computing centers, a storage pool is usually configured for the intelligent training cluster. Two types of storage clusters are configured in the storage pool: performance storage and capacity storage. Among them, performance storage is used to provide high-speed data access, checkpoint data and model data storage and reading during the training process. Capacity storage is used to save original training data and archive data. Figure 1 shown.

[0050] The size of the storage pool space is usually estimated based on the scale of intelligent computing power based on experience, such as "2PB storage for every 100 PFLOPS of intelligent computing power", or based on the number of AI accelerator cards in the intelligent computing cluster, such as "1PB of high-performance file storage and 2PB of ordinary performance file storage for cards below 1,000; 2PB of high-performance file storage and 4PB of ordinary performance file storage for cards above 1,000, etc." This solution of determining the size of the storage pool space based on experience often results in waste of storage resources due to over-allocation or insufficient storage resources leading to capacity expansion later.

[0051] In order to alleviate the technical problems existing in the related art, the embodiments of the present application provide a storage capacity calculation method, device, equipment, storage medium and computer program product. The method divides the task-related data of artificial intelligence training into multiple data categories, performs modeling and calculation respectively, obtains the amount of training data, the amount of original data, the amount of model parameter data, the amount of staged model state data and the amount of optimizer data, and calculates the total task data amount of the model based on these data amounts, thereby calculating the storage capacity size that needs to be configured for the first training cluster during the model training process based on the total task data amount. It can be seen that the solution of the present application realizes the scientific prediction of storage capacity, which helps to improve resource utilization.

[0052] To facilitate understanding of this embodiment, a storage capacity calculation method disclosed in an embodiment of the present application is first introduced in detail. The execution subject of the storage capacity calculation method provided in the embodiment of the present application is generally an electronic device with certain computing capabilities, and the electronic device includes, for example, a terminal device or a server or other processing device. In some possible implementations, the storage capacity calculation method can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0053] See also Figure 2 FIG. 1 is a flowchart of a storage capacity calculation method provided in an embodiment of the present application, wherein the method comprises the following steps:

[0054] Step 201: Based on the computing resource parameters of the first training cluster and the training parameters required for the model deployed in the first training cluster to complete a single training task, predict the amount of training data and the amount of original data required for the model to complete a single training task; and, based on the training parameters, predict the amount of model parameter data, the amount of staged model state data, and the amount of data of the optimizer used to optimize the parameters of the model.

[0055] In this embodiment, a single training task refers to the process of training a model to convergence using a preset training sample set. The amount of training data refers to the storage space required to complete a single training task in the first training cluster, and the amount of raw data refers to the storage space required for training data formed after the raw data is cleaned and tokenized during the intelligent training process.

[0056] It should be understood that in the process of intelligent training, the amount of training data is related to the number of tokens processed by the training cluster and the number of bytes occupied by the tokens. The number of tokens that the training cluster can process is related to the intelligent computing power of the cluster, the training duration, and the number of parameters of the model. Therefore, in this embodiment, the computing resource parameters are used to characterize the computing power of the first training cluster. In the application, the computing resource parameters include but are not limited to the number of processors configured in the first training cluster and the effective computing power of a single processor. The training parameters include but are not limited to the training duration of a single training task, the number of model parameters of the model, the number of bytes occupied by the corpus, and the number of corpora processed using computing power per unit parameter.

[0057] On this basis, the calculation of the amount of training data and the amount of original data in step 201 may include the following steps:

[0058] Calculate the product of the effective computing power of a single processor, the number of processors, and the training time of a single training task to obtain a first operation result, where the first operation result is used to characterize the total number of floating-point operations required for the model to complete a single training task;

[0059] The quotient of the first operation result and the model parameter quantity is obtained to obtain a second operation result, and the second operation result is used to represent the number of floating-point operations corresponding to a single model parameter;

[0060] The product of the second operation result, the number of bytes occupied by the corpus and the number of corpora processed by the unit parameter amount using computing power is calculated to obtain the amount of training data, and the quotient of the amount of training data and the preset cleaning coefficient is calculated to obtain the amount of original data.

[0061] The amount of training data and the amount of original data expressed in the formula are as follows:

[0062]

[0063] in:

[0064] D1: the amount of training data for a single training task, unit: Byte;

[0065] GPU: the number of GPU cards configured in the training cluster;

[0066] CP / c : The effective computing power of a single GPU card (the number of floating-point operations per second FLOPS), unit: TFLOPS;

[0067] T: training duration, unit: day;

[0068] P: model parameter quantity, unit: Billion (10^9);

[0069] T B : The number of bytes occupied by the token, in Byte;

[0070] C1: The number of tokens processed by the unit parameter using computing power, unit: flops / parameter.

[0071] In the process of intelligent training, the original data is cleaned, segmented and tokenized to form training data. The original data volume can be deduced based on the training data and the data cleaning coefficient. The specific formula can be:

[0072]

[0073] in:

[0074] D0: the amount of raw data for a single training task, unit: Byte;

[0075] C2: Cleaning coefficient. The data after the original data is cleaned is the training data.

[0076] In this embodiment, the model parameter data volume refers to the size of the storage space required to store the model parameters. The optimizer is used to optimize the model parameters, so the optimizer data volume refers to the size of the storage space occupied by the optimizer. The staged model state (CheckPoint) data volume refers to the size of the storage space required to save the CheckPoint of the intermediate state of model training.

[0077] In some optional embodiments, the calculation of the model parameter data volume, the staged model state data volume, and the optimizer data volume may include the following steps:

[0078] From the training parameters, obtain the model parameter quantity and the number of bytes occupied by the model parameters;

[0079] Calculate the product of the model parameter quantity and the number of bytes occupied by the model parameters to obtain the model parameter data quantity;

[0080] The product of the model parameter data volume and the preset stage-by-stage model state coefficient is calculated to obtain the data volume of the optimizer; and the model parameter data volume is used as the stage-by-stage model state data volume.

[0081] To expand on this, the calculation formula for the amount of model parameter data is:

[0082] D p =P×P B

[0083] in:

[0084] D p : Model parameter data volume, unit: Byte;

[0085] P: model parameter quantity, unit: Billion (10^9);

[0086] P B : The number of bytes occupied by model parameters, in Byte.

[0087] The calculation formula for the amount of stage model state data is:

[0088] D cp ≈D p =P×P B

[0089] in:

[0090] D cp : CheckPoint data volume, unit: Byte.

[0091] The formula for calculating the data volume of the optimizer is:

[0092] D op =D cp ×C3

[0093] D op : Optimizer data volume, unit: Byte;

[0094] C3: CheckPoint coefficient (expansion coefficient).

[0095] Step 202: Calculate the task data volume of the model during the training process based on the amount of training data, the amount of original data, the amount of model parameter data, the amount of staged model state data, and the amount of optimizer data.

[0096] In this embodiment, when the model is a multimodal model, since the multimodal model can process multiple types of model tasks at the same time, such as the multimodal model can process large natural language processing model tasks, large computer vision model tasks and / or large automatic speech recognition model tasks, etc., the task data volume of the multimodal model is the sum of the task data volumes of multiple types of model tasks processed by the model at the same time. When the model is a unimodal model, the model can only process one type of model task at the same time, so the task data volume of the model is the task data volume of the model task of that type.

[0097] Taking a certain training task as an example, in this embodiment, the data of the model training task includes original data, training data, optimizer data, CheckPoint data and model parameter data. Based on these data, the training data volume, original data volume, model parameter data volume, staged model state data volume and optimizer data volume are calculated respectively. Therefore, the task data volume of this type of model task is the weighted sum of the training data volume, original data volume, model parameter data volume, staged model state data volume and optimizer data volume. When expressed in formula, the task data volume of the model task can be expressed as:

[0098]

[0099] in:

[0100] D t : Task data volume, unit: Byte;

[0101] C4: reuse coefficient of original data for training tasks;

[0102] N: The number of CheckPoints archived for each training task;

[0103] t: CheckPoint data is saved every t hours during training.

[0104] When the model is a multimodal model, the task data volume of the model task can be expressed as:

[0105] D T =D NLP ×x+D CV ×y+D ASR ×

[0106] in:

[0107] D T : Total data volume of the multimodal large model, unit: Byte;

[0108] D NLP : The amount of data for large natural language processing model tasks;

[0109] D CV : The amount of data for large computer vision model tasks;

[0110] D ASR : The amount of data for large model tasks of automatic speech recognition;

[0111] x, y, z: The number of tasks for the three types of large model configurations.

[0112] Step 203: Based on the amount of task data, calculate the storage capacity required for model training.

[0113] In this embodiment, the storage capacity required for multimodal large model training is calculated based on the task data volume, metadata occupancy, and the upper limit of available storage device space. In an optional embodiment, step 203 may include the following steps:

[0114] Calculate the product of the task data volume and the preset metadata ratio to obtain a third operation result;

[0115] The quotient of the third operation result and the upper limit ratio of the storage device available in the first training cluster is calculated to obtain the storage capacity.

[0116] When expressed in formula, the storage capacity can be expressed as:

[0117]

[0118] in:

[0119] S B : Storage capacity, unit: Byte;

[0120] D T : Total data volume of the multimodal large model, unit: Byte;

[0121] m: metadata ratio;

[0122] SWL: The upper limit of available storage capacity.

[0123] In the solution provided in this embodiment, the task-related data of artificial intelligence training is divided into multiple data categories, and modeling and calculation are performed separately to obtain the training data volume, the original data volume, the model parameter data volume, the staged model state data volume and the optimizer data volume, and the total task data volume of the model is calculated based on these data volumes, so as to calculate the storage capacity size that needs to be configured for the first training cluster during the model training process based on the total task data volume. It can be seen that the solution of this application realizes the scientific prediction of storage capacity, which helps to improve resource utilization.

[0124] In one or more embodiments, after calculating the storage capacity required for model training based on the task data volume, the following steps may also be included:

[0125] Calculate the computing-storage ratio based on storage capacity and computing resource parameters;

[0126] The computing-storage ratio and the computing resource parameters of the second training cluster are used to calculate the storage capacity required for model training deployed in the second training cluster.

[0127] The industry generally uses "how many PB of storage are configured for every 100P of intelligent computing power" to describe the computing-storage ratio. Based on the above calculations, the computing-storage ratio in the multi-modal large model training scenario can be obtained:

[0128]

[0129] in:

[0130] PB / 100PFLOPS : Computing-storage ratio, storage size configured per 100P intelligent computing power, unit: PByte;

[0131] S B : Storage capacity, unit: Byte;

[0132] GPU: the number of GPU cards configured in the training cluster;

[0133] CP / c : The effective computing power of a single GPU card (the number of floating-point operations per second FLOPS, unit: TFLOPS).

[0134] It should be understood that by calculating the compute-to-storage ratio, the storage capacity that needs to be configured for the second training cluster can be determined in proportion according to the compute-to-storage ratio and in combination with the computing parameters of the second training cluster, thereby saving the amount of computation required to calculate the storage capacity by calculating the amount of training data, the amount of original data, the amount of model parameter data, the amount of staged model state data, and the amount of optimizer data, thereby improving computing efficiency.

[0135] During the training of a large multimodal model, the training data, CheckPoint data, and some original data are accessed frequently, and high-performance storage devices need to be configured for them. Most of the original data, archived CheckPoint data, and model parameter data are accessed less frequently, and the amount of this data is large, so capacity storage devices can be configured. Therefore, hierarchical storage is set up for these two situations. When it is implemented specifically, based on the computing resource parameters of the first training cluster, the amount of training data and the amount of original data for the model deployed in the training cluster to complete a single training task are predicted; and based on the model parameter amount of the model, the amount of model parameter data, the amount of staged model state data, and the amount of optimizer data used to optimize the model parameters are predicted. The following steps can also be included:

[0136] Calculate high-performance storage capacity based on the amount of training data, raw data, staged model state data, and optimizer data; and,

[0137] The capacity storage capacity is calculated based on the amount of original data, the amount of model parameters of the model, the amount of staged model state data, and the amount of optimizer data.

[0138] The formula for high-performance storage capacity and capacity storage capacity is as follows:

[0139] PS=D1+D0×C5+Training_CP

[0140] CS=D0×(1-C5)+Archiving_CP+D p

[0141] in:

[0142] PS: high-performance storage capacity, unit: PByte;

[0143] CS: Capacity storage capacity, unit: PByte;

[0144] C5: The proportion of raw data with high access frequency;

[0145] Archiving_CP: Archiving CheckPoint data, unit: PByte;

[0146] Training_CP: training CheckPoint data, unit: PByte;

[0147] D p : Model parameter data.

[0148] in,

[0149] Archiving_CP=(D cp +D op )×N.

[0150] The present application also provides a storage capacity calculation device for executing the storage capacity calculation method provided in any of the above embodiments. Figure 3 As shown, the device comprises:

[0151] A prediction module 31 is used to predict the amount of training data and the amount of original data required for the model to complete a single training task based on the computing resource parameters of the first training cluster and the training parameters required for the model deployed in the first training cluster to complete the single training task; and, based on the training parameters, predict the amount of model parameter data, the amount of staged model state data, and the amount of data of the optimizer used to optimize the parameters of the model;

[0152] A first calculation module 32, used to calculate the task data volume of the model during the training process based on the training data volume, the original data volume, the model parameter data volume, the staged model state data volume and the optimizer data volume;

[0153] The second calculation module 33 is used to calculate the storage capacity required for the model training based on the task data volume.

[0154] In one or more embodiments, the device is also used to:

[0155] After calculating the storage capacity required for model training based on the task data volume, the computing-storage ratio is calculated based on the storage capacity and the computing resource parameters;

[0156] The computing-storage ratio and the computing resource parameters of the second training cluster are used to calculate the storage capacity required for model training deployed in the second training cluster.

[0157] In one or more embodiments, the device is also used to:

[0158] Based on the computing resource parameters of the first training cluster, predict the amount of training data and the amount of original data required for a model deployed in the training cluster to complete a single training task; and, based on the amount of model parameters of the model, predict the amount of model parameter data, the amount of staged model state data, and the amount of data of an optimizer for optimizing the parameters of the model, and then calculate the high-performance storage capacity based on the amount of training data, the amount of original data, the amount of staged model state data, and the amount of data of the optimizer; and,

[0159] The capacity-based storage capacity is calculated based on the original data volume, the model parameter volume of the model, the staged model state data volume and the optimizer data volume.

[0160] In one or more embodiments, the prediction module 31 is used to:

[0161] From the computing resource parameters, the number of processors configured in the first training cluster and the effective computing power of a single processor are obtained; and from the training parameters, the training duration of the single training task, the model parameter quantity of the model, the number of bytes occupied by the corpus, and the number of corpuses processed using computing power per unit parameter quantity are obtained;

[0162] Calculate the product of the effective computing power of the single processor, the number of the processors, and the training duration of the single training task to obtain a first operation result, where the first operation result is used to characterize the total number of floating-point operations required for the model to complete the single training task;

[0163] Taking the quotient of the first operation result and the model parameter quantity, a second operation result is obtained, where the second operation result is used to represent the number of floating-point operations corresponding to a single model parameter;

[0164] The product of the second operation result, the number of bytes occupied by the corpus and the number of corpora processed by the unit parameter amount using computing power is calculated to obtain the training data volume, and the quotient of the training data volume and a preset cleaning coefficient is calculated to obtain the original data volume.

[0165] In one or more embodiments, the prediction module 31 is used to:

[0166] Obtaining the model parameter quantity and the number of bytes occupied by the model parameters of the model from the training parameters;

[0167] Calculate the product of the model parameter amount and the number of bytes occupied by the model parameter to obtain the model parameter data amount;

[0168] The product of the model parameter data volume and the preset stage-by-stage model state coefficient is calculated to obtain the data volume of the optimizer; and the model parameter data volume is used as the stage-by-stage model state data volume.

[0169] In one or more embodiments, the second calculation module 33 is used to:

[0170] Calculating the product of the task data volume and the preset metadata ratio to obtain a third operation result;

[0171] The storage capacity is obtained by calculating a quotient of the third operation result and an upper limit ratio of storage devices of the first training cluster.

[0172] In one or more embodiments, the device is used to:

[0173] Based on the storage capacity, calculating the total number of floating-point operations of the first training cluster;

[0174] The calculation-to-memory ratio is obtained by calculating the quotient of the total number of floating-point operations, the number included in the computing resource parameter, and the effective computing power of a single processor.

[0175] The storage capacity calculation device provided in the embodiment of the present application and the storage capacity calculation method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented therein.

[0176] The present application also provides an electronic device to perform the above storage capacity calculation method. Figure 4 It shows a schematic diagram of an electronic device provided by some embodiments of the present application. Figure 4 As shown, the electronic device 8 includes: a processor 800, a memory 801, a bus 802 and a communication interface 803, wherein the processor 800, the communication interface 803 and the memory 801 are connected via the bus 802; the memory 801 stores a computer program that can be run on the processor 800, and when the processor 800 runs the computer program, the storage capacity calculation method provided in any of the aforementioned embodiments of the present application is executed.

[0177] The memory 801 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the device network element and at least one other network element is realized through at least one communication interface 803 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used.

[0178] The bus 802 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. The memory 801 is used to store programs, and the processor 800 executes the program after receiving an execution instruction. The storage capacity calculation method disclosed in any implementation of the embodiment of the present application may be applied to the processor 800, or implemented by the processor 800.

[0179] The processor 800 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 800. The above processor 800 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a readily available programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be executed. The software module can be located in a mature storage medium in the field such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 801, and the processor 800 reads the information in the memory 801 and completes the steps of the above method in combination with its hardware.

[0180] The electronic device provided in the embodiment of the present application and the storage capacity calculation method provided in the embodiment of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, operated or implemented therein.

[0181] The present application also provides a computer-readable storage medium corresponding to the storage capacity calculation method provided in the above embodiment. Figure 5 The computer-readable storage medium shown is a CD 30 on which a computer program (ie, a program product) is stored. When the computer program is executed by a processor, the storage capacity calculation method provided by any of the aforementioned embodiments will be executed.

[0182] It should be noted that examples of the computer-readable storage medium may also include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other optical or magnetic storage media, which are not listed here one by one.

[0183] The computer-readable storage medium provided in the above-mentioned embodiments of the present application and the storage capacity calculation method provided in the embodiments of the present application are based on the same inventive concept and have the same beneficial effects as the methods adopted, run or implemented by the application programs stored therein.

[0184] It should be noted that:

[0185] In the description provided herein, a large number of specific details are described. However, it is understood that the embodiments of the present application can be practiced without these specific details. In some instances, well-known structures and technologies are not shown in detail so as not to obscure the understanding of this description.

[0186] Similarly, it should be understood that in order to streamline the present application and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the present application, the various features of the present application are sometimes grouped together into a single embodiment, figure, or description thereof. However, the disclosed method should not be interpreted as reflecting the following schematic diagram: the claimed application requires more features than the features clearly stated in each claim. More specifically, as reflected in the claims below, the inventive aspects are less than all the features of the single embodiment disclosed above. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present application.

[0187] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present application and form different embodiments. For example, in the claims below, any one of the claimed embodiments may be used in any combination.

[0188] The above is only a preferred specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A storage capacity calculation method, characterized in that: include: Based on the computing resource parameters of the first training cluster and the training parameters required by the model deployed in the first training cluster to complete a single training task, predict the amount of training data and the amount of original data required by the model to complete the single training task; and based on the training parameters, predict the amount of model parameter data, the amount of staged model state data, and the amount of data of the optimizer used to optimize the parameters of the model; Calculate the task data volume of the model during the training process based on the training data volume, the original data volume, the model parameter data volume, the staged model state data volume and the optimizer data volume; Based on the amount of task data, calculate the storage capacity required for model training.

2. The method according to claim 1, characterized in that: After calculating the storage capacity required for model training based on the task data volume, the method further includes: Calculating a computing-storage ratio based on the storage capacity and the computing resource parameters; The computing-storage ratio and the computing resource parameters of the second training cluster are used to calculate the storage capacity required for model training deployed in the second training cluster.

3. The method according to claim 1, characterized in that Based on the computing resource parameters of the first training cluster, the amount of training data and the amount of original data of the model deployed in the training cluster to complete a single training task are predicted; and based on the model parameter amount of the model, the amount of model parameter data, the amount of staged model state data, and the amount of data of the optimizer for optimizing the parameters of the model are predicted, further comprising: Calculating high-performance storage capacity based on the amount of training data, the amount of original data, the amount of staged model state data, and the amount of optimizer data; and The capacity-based storage capacity is calculated based on the original data volume, the model parameter volume of the model, the staged model state data volume and the optimizer data volume.

4. The method according to claim 1, characterized in that: Predicting the amount of training data and the amount of original data required for the model to complete the single training task based on the computing resource parameters of the first training cluster and the training parameters required for the model deployed in the first training cluster to complete the single training task includes: From the computing resource parameters, the number of processors configured in the first training cluster and the effective computing power of a single processor are obtained; and from the training parameters, the training duration of the single training task, the model parameter quantity of the model, the number of bytes occupied by the corpus, and the number of corpuses processed using computing power per unit parameter quantity are obtained; Calculate the product of the effective computing power of the single processor, the number of the processors, and the training duration of the single training task to obtain a first operation result, where the first operation result is used to characterize the total number of floating-point operations required for the model to complete the single training task; Taking the quotient of the first operation result and the model parameter quantity, a second operation result is obtained, where the second operation result is used to represent the number of floating-point operations corresponding to a single model parameter; The product of the second operation result, the number of bytes occupied by the corpus and the number of corpora processed by the unit parameter amount using computing power is calculated to obtain the training data volume, and the quotient of the training data volume and a preset cleaning coefficient is calculated to obtain the original data volume.

5. The method according to claim 1, characterized in that Based on the training parameters, the amount of model parameter data, the amount of staged model state data, and the amount of optimizer data for optimizing the parameters of the model are predicted, including: Obtaining the model parameter quantity and the number of bytes occupied by the model parameters of the model from the training parameters; Calculate the product of the model parameter amount and the number of bytes occupied by the model parameter to obtain the model parameter data amount; The product of the model parameter data volume and the preset stage-by-stage model state coefficient is calculated to obtain the data volume of the optimizer; and the model parameter data volume is used as the stage-by-stage model state data volume.

6. The method according to claim 1, characterized in that Based on the task data volume, the storage capacity required for model training is calculated, including: Calculating the product of the task data volume and the preset metadata ratio to obtain a third operation result; The storage capacity is obtained by calculating a quotient of the third operation result and an upper limit ratio of storage devices of the first training cluster.

7. The method according to claim 2, characterized in that Calculating the computing-storage ratio based on the storage capacity and the computing resource parameter includes: Based on the storage capacity, calculating the total number of floating-point operations of the first training cluster; The calculation-to-memory ratio is obtained by calculating the quotient of the total number of floating-point operations, the number included in the computing resource parameter, and the effective computing power of a single processor.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, characterized in that: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images