Method and device for determining large model training strategy and electronic equipment

Abstract

The invention discloses a method and device for determining a large model training strategy and electronic equipment. The method comprises the steps of obtaining hardware parameter information, model parameter information and training parameter information corresponding to a to-be-trained model; determining a first training strategy based on the hardware parameter information, the model parameter information and the training parameter information by adopting a first training framework, and determining a second training strategy based on the hardware parameter information, the model parameter information and the training parameter information by adopting a second training framework, the training algorithms corresponding to the first training framework and the second training framework are different; and determining a target training strategy corresponding to the to-be-trained model from the first training strategy and the second training strategy according to a preset training step number and a preset data set. The technical problem that the accuracy of the determined training strategy is low when the training strategy of the model is determined based on expert experience in the related technology is solved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and more specifically, to a method, device and electronic device for determining a large model training strategy. Background Art

[0002] With the continuous development of large models, the size of models is getting larger and larger, and the training data is also increasing. The training of large models often faces problems such as the model is too large to fit, and the training time is too long due to too much data. Traditional training optimization needs to rely on expert experience, and multiple experiments and adjustments on graphics cards, data, models, etc. are required to determine the corresponding training strategy, which has low accuracy and efficiency.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present application provide a method, device and electronic device for determining a large model training strategy, so as to at least solve the technical problem in the related art that the training strategy of the model is determined based on expert experience, and the accuracy of the determined training strategy is low.

[0005] According to one aspect of an embodiment of the present application, a method for determining a large model training strategy is provided, including: obtaining hardware parameter information, model parameter information and training parameter information corresponding to a model to be trained; using a first training framework to determine a first training strategy based on the hardware parameter information, model parameter information and training parameter information, and using a second training framework to determine a second training strategy based on the hardware parameter information, model parameter information and training parameter information, wherein the first training framework and the second training framework correspond to different training algorithms; determining a target training strategy corresponding to the model to be trained from the first training strategy and the second training strategy based on a preset number of training steps and a preset data set.

[0006] Furthermore, the first training framework is a training framework that supports fine-tuning training. Using the first training framework to determine the first training strategy based on hardware parameter information, model parameter information and training parameter information includes: configuring operator optimization and recalculation of the first training framework to be turned on; determining the total available video memory based on the hardware parameter information; obtaining an arrangement order of multiple candidate training strategies in the first training framework, and based on the arrangement order, determining a total training video memory corresponding to a candidate training strategy among the multiple candidate training strategies based on the model parameter information and the training parameter information; determining the first training strategy from multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategy among the multiple candidate training strategies.

[0007] Furthermore, determining the first training strategy from multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies in the multiple candidate training strategies includes: if the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, configuring the recalculation to be closed, and re-determining the total training video memory corresponding to the current candidate training strategy based on the model parameter information and the training parameter information; if the total training video memory corresponding to the re-determined current candidate training strategy is less than or equal to the total available video memory, using the current candidate training strategy as the first training strategy.

[0008] Furthermore, determining a first training strategy from multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies in the multiple candidate training strategies includes: if the total training video memory corresponding to the current candidate training strategy is greater than the total available video memory, then based on the arrangement order, determining the total training video memory corresponding to the next candidate training strategy of the current candidate training strategy based on the model parameter information and the training parameter information, until the total training video memory corresponding to the candidate training strategies is less than or equal to the total available video memory, thereby obtaining the first training strategy.

[0009] Furthermore, the second training framework is a training framework that supports full-parameter training. The second training framework is used to determine the second training strategy based on hardware parameter information, model parameter information and training parameter information, including: configuring the sequence parallelism of the second training framework to be on, and configuring the recalculation to be off; determining the total available video memory based on the hardware parameter information; configuring the pipeline parallelism, tensor parallelism and data parallelism of the second training framework based on preset configuration information to obtain multiple candidate training strategies of the second training framework; determining the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies based on the model parameter information and the training parameter information; determining the second training strategy from the multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies.

[0010] Furthermore, based on the preset configuration information, the pipeline parallelism, tensor parallelism and data parallelism of the second training framework are configured to obtain multiple candidate training strategies for the second training framework, including: configuring pipeline parallelism to a first preset value, and configuring tensor parallelism to sequentially take values ​​according to the preset values ​​in the first preset value set, to obtain a first candidate training strategy set; configuring tensor parallelism to a second preset value, and configuring pipeline parallelism to sequentially take values ​​according to the preset values ​​in the second preset value set, to obtain a second candidate training strategy set; configuring tensor parallelism to a third preset value, and configuring pipeline parallelism to sequentially take values ​​according to the preset values ​​in the second preset value set, to obtain a third candidate training strategy set; configuring tensor parallelism to a third preset value, configuring pipeline parallelism to a third preset value, and configuring the recalculated states in sequence based on multiple preset states to obtain a fourth candidate training strategy set; determining multiple candidate training strategies for the second training framework based on the first candidate training strategy set, the second candidate training strategy set, the third candidate training strategy set and the fourth candidate training strategy set.

[0011] Furthermore, determining the second training strategy from the multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies includes: if the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, then using the current candidate training strategy as the second training strategy.

[0012] Furthermore, based on a preset number of training steps and a preset data set, determining a target training strategy corresponding to the model to be trained from the first training strategy and the second training strategy includes: performing model training on the model to be trained according to the first training strategy based on the preset number of training steps and the preset data set to obtain a throughput corresponding to the first training strategy; performing model training on the model to be trained according to the second training strategy based on the preset number of training steps and the preset data set to obtain a throughput corresponding to the second training strategy; and taking the training strategy with a large throughput as the target training strategy.

[0013] According to another aspect of an embodiment of the present application, a method for generating a target model is also provided, including: obtaining training data; training a large model to be trained using a target training strategy and the training data to obtain a trained target large model, wherein the target training strategy is obtained using any of the above-mentioned methods for determining the large model training strategy.

[0014] According to another aspect of an embodiment of the present application, a method for determining a large model training strategy is also provided, including: obtaining hardware parameter information, model parameter information and training parameter information corresponding to the model to be trained uploaded by the client; using a first training framework in the cloud server to determine a first training strategy based on the hardware parameter information, model parameter information and training parameter information, and using a second training framework to determine a second training strategy based on the hardware parameter information, model parameter information and training parameter information, wherein the training algorithms corresponding to the first training framework and the second training framework are different; determining a target training strategy corresponding to the model to be trained from the first training strategy and the second training strategy based on a preset number of training steps and a preset data set; and feeding back the target training strategy to the client.

[0015] According to another aspect of an embodiment of the present application, a device for determining a large model training strategy is also provided, including: a first acquisition unit, used to acquire hardware parameter information, model parameter information and training parameter information corresponding to the model to be trained; a first determination unit, used to determine a first training strategy based on the hardware parameter information, model parameter information and training parameter information using a first training framework, and to determine a second training strategy based on the hardware parameter information, model parameter information and training parameter information using a second training framework, wherein the training algorithms corresponding to the first training framework and the second training framework are different; a second determination unit, used to determine a target training strategy corresponding to the model to be trained from the first training strategy and the second training strategy based on a preset number of training steps and a preset data set.

[0016] Furthermore, the first determination unit includes: a first configuration subunit, used to configure the operator optimization and recalculation of the first training framework to be in an on state; a first determination subunit, used to determine the total available video memory based on the hardware parameter information; a second determination subunit, used to obtain the arrangement order of multiple candidate training strategies in the first training framework, and based on the arrangement order, determine the total training video memory corresponding to the candidate training strategies in the multiple candidate training strategies according to the model parameter information and the training parameter information; a third determination subunit, used to determine the first training strategy from multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies in the multiple candidate training strategies.

[0017] Furthermore, the third determination subunit includes: a first determination module, which is used to configure the recalculation to be closed if the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, and re-determine the total training video memory corresponding to the current candidate training strategy based on the model parameter information and the training parameter information; a second determination module, which is used to use the current candidate training strategy as the first training strategy if the total training video memory corresponding to the re-determined current candidate training strategy is less than or equal to the total available video memory.

[0018] Furthermore, the third determination subunit includes: a third determination module, which is used to determine the total training video memory corresponding to the next candidate training strategy of the current candidate training strategy based on the arrangement order, according to the model parameter information and the training parameter information, if the total training video memory corresponding to the current candidate training strategy is greater than the total available video memory, until the total training video memory corresponding to the candidate training strategy is less than or equal to the total available video memory, so as to obtain the first training strategy.

[0019] Furthermore, the first determination unit also includes: a second configuration subunit, used to configure the sequence parallelism of the second training framework to be in an on state, and to configure the recalculation to be in an off state; a fourth determination subunit, used to determine the total available video memory based on the hardware parameter information; a third configuration subunit, used to configure the pipeline parallelism, tensor parallelism and data parallelism of the second training framework based on preset configuration information, to obtain multiple candidate training strategies for the second training framework; a fifth determination subunit, used to determine the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies based on the model parameter information and the training parameter information; a sixth determination subunit, used to determine the second training strategy from multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies.

[0020] Furthermore, the third configuration subunit includes: a first configuration module, which is used to configure the pipeline parallelism to a first preset value, and configure the tensor parallelism to sequentially take values ​​according to the preset values ​​in the first preset value set, to obtain a first candidate training strategy set; a second configuration module, which is used to configure the tensor parallelism to a second preset value, and configure the pipeline parallelism to sequentially take values ​​according to the preset values ​​in the second preset value set, to obtain a second candidate training strategy set; a third configuration module, which is used to configure the tensor parallelism to a third preset value, and configure the pipeline parallelism to sequentially take values ​​according to the preset values ​​in the second preset value set, to obtain a third candidate training strategy set; a fourth configuration module, which is used to configure the tensor parallelism to a third preset value, configure the pipeline parallelism to a third preset value, and sequentially configure the recalculated state based on multiple preset states to obtain a fourth candidate training strategy set; a fourth determination module, which is used to determine multiple candidate training strategies of the second training framework based on the first candidate training strategy set, the second candidate training strategy set, the third candidate training strategy set and the fourth candidate training strategy set.

[0021] Furthermore, the sixth determination subunit includes: a fifth determination module, configured to use the current candidate training strategy as the second training strategy if the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory.

[0022] Furthermore, the second determination unit includes: a first processing subunit, used to perform model training on the training model according to a first training strategy based on a preset number of training steps and a preset data set, so as to obtain a throughput corresponding to the first training strategy; a second processing subunit, used to perform model training on the training model according to a second training strategy based on a preset number of training steps and a preset data set, so as to obtain a throughput corresponding to the second training strategy; and a seventh determination subunit, used to use a training strategy with a large throughput as a target training strategy.

[0023] According to another aspect of an embodiment of the present application, a device for generating a target model is also provided, including: a second acquisition unit, used to acquire training data; a training unit, used to train a large model to be trained using a target training strategy and the training data to obtain a trained target large model, wherein the target training strategy is obtained using any of the above-mentioned methods for determining the large model training strategy.

[0024] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising: a memory storing an executable program; and a processor for running the program, wherein when the program is running, any one of the above-mentioned methods for determining a large model training strategy is executed.

[0025] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is also provided, wherein the storage medium stores a program, wherein when the program is running, a method for determining a large model training strategy is controlled to execute any one of the above-mentioned device where the storage medium is located.

[0026] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a computer program or instructions, which implements any of the above-mentioned methods for determining a large model training strategy when executed by a processor.

[0027] In an embodiment of the present application, by obtaining hardware parameter information, model parameter information and training parameter information corresponding to the model to be trained; using a first training framework to determine a first training strategy based on the hardware parameter information, model parameter information and training parameter information, and using a second training framework to determine a second training strategy based on the hardware parameter information, model parameter information and training parameter information, wherein the training algorithms corresponding to the first training framework and the second training framework are different; according to a preset number of training steps and a preset data set, a target training strategy corresponding to the model to be trained is determined from the first training strategy and the second training strategy, thereby realizing automatic determination of a better training strategy based on actual hardware configuration, model parameters, etc., and automatic strategy verification based on a preset number of training steps and a preset data set, thereby improving the accuracy of the determined strategy, effectively saving time and cost, and achieving the purpose of improving the training speed of large models, thereby achieving the technical effect of improving the accuracy and efficiency of training strategy selection, and then solving the technical problem of low accuracy of the determined training strategy in the related technology that the training strategy of the model is determined based on expert experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0029] Figure 1 is a schematic diagram of a computer terminal provided according to Embodiment 1 of the present application;

[0030] Figure 2 It is a flowchart of a method for determining a large model training strategy according to the first embodiment of the present application;

[0031] Figure 3 It is a schematic diagram of an optional process of determining a target training strategy provided in Embodiment 1 of the present application;

[0032] Figure 4 is a flow chart of a method for generating a target model according to Embodiment 2 of the present application;

[0033] Figure 5 It is a flowchart of a method for determining a large model training strategy according to Embodiment 3 of the present application;

[0034] Figure 6 It is a schematic diagram of a device for determining a large model training strategy according to the fourth embodiment of the present application;

[0035] Figure 7 is a schematic diagram of a device for generating a target model according to Embodiment 5 of the present application;

[0036] Figure 8This is a structural block diagram of an electronic device provided according to Embodiment 6 of the present application. DETAILED DESCRIPTION

[0037] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0038] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0039] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards in the relevant regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0040] First, some nouns or terms that appear in the description of the embodiments of the present application are subject to the following explanations:

[0041] Tensor parallelism: The full English name is Tensor Parallelism, abbreviated as TP.

[0042] Pipeline parallelism: The full English name is Pipeline Parallelism, abbreviated as PP.

[0043] Data parallelism: The full English name is Data Parallelism, abbreviated as DP.

[0044] Sequence parallelism: The full English name is Sequence Parallelism, abbreviated as SP.

[0045] Context Parallelism: The full English name is Context Parallelism, abbreviated as CP.

[0046] zero: A solution to remove data redundancy, generally including three stages: zero1, zero2, and zero3.

[0047] Full parameter training: When training a large model, all model parameters will be updated.

[0048] Low-Rank Adaptation (LoRA): When training a large model, a small number of parameters are added for training, and the original model weights remain frozen.

[0049] Example 1

[0050] According to an embodiment of the present application, a method for determining a large model training strategy is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0051] The method embodiment provided in the first embodiment of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Figure 1 The hardware structure block diagram of a computer terminal (or mobile device) for implementing a method for determining a large model training strategy is shown. Figure 1 As shown, the computer terminal (or mobile device) 10 may include a processor set 102 (the processor set 102 may include but is not limited to a processing device such as a microprocessor (Microcontroller Unit, referred to as MCU) or a programmable logic device (Field Programmable Gate Array, referred to as FPGA), and the processor set 102 may include a processor set, Figure 1 102a, 102b, ..., 102n are used to illustrate), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a Universal Serial Bus (USB) port (which may be included as one of the ports of the BUS bus), a network interface, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 1More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0052] It should be noted that the one or more processors 102 and / or other data processing circuits described above may generally be referred to herein as "data processing circuits". The data processing circuits may be embodied in whole or in part as software, hardware, firmware, or any other combination thereof. In addition, the data processing circuits may be a single independent processing module, or may be integrated in whole or in part into any of the other components in the computer terminal 10 (or mobile device).

[0053] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the method for determining the large model training strategy in the embodiment of the present application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, that is, the method for determining the large model training strategy described above is realized. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0054] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer terminal 10. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0055] The display may be a touch screen type liquid crystal display that enables a user to interact with a user interface of the computer terminal 10 (or mobile device).

[0056] With the continuous development of large models, the size of models is getting larger and larger, and the training data is also increasing. The training of large models often faces problems such as the model is too large to fit, and the training time is too long due to too much data. Traditional training optimization needs to rely on expert experience, and multiple experiments and adjustments on graphics cards, data, models, etc. can determine the corresponding training strategy, which has low accuracy and efficiency. With more and more open source models, how to automatically and efficiently complete training according to actual needs to reduce costs and save time has become an urgent problem for various large model manufacturers.

[0057] In the above technical background, the present application provides Figure 2 The method for determining the large model training strategy is shown. Figure 2 This is a flow chart of a method for determining a large model training strategy according to the first embodiment of the present application. The method includes:

[0058] Step S201, obtaining hardware parameter information, model parameter information and training parameter information corresponding to the model to be trained.

[0059] Optionally, the hardware corresponding to the model to be trained refers to a computing device used to train a large model, the hardware parameter information includes the number of graphics cards, the size of video memory, the bandwidth of video memory, the communication bandwidth, etc., the model parameter information includes the model size, the number of layers, the number of attention heads, etc., and the training parameter information includes the batch size, the sequence length, etc. For example, the user inputs the hardware parameter information, the model parameter information, and the training parameter information corresponding to the model to be trained, and the large model training strategy determination system obtains this information.

[0060] Step S202, using a first training framework to determine a first training strategy based on hardware parameter information, model parameter information and training parameter information, and using a second training framework to determine a second training strategy based on hardware parameter information, model parameter information and training parameter information, wherein the training algorithms corresponding to the first training framework and the second training framework are different.

[0061] Optionally, the first training framework and the second training framework can select a mainstream large model training framework. After obtaining the hardware parameter information, model parameter information and training parameter information corresponding to the model to be trained, the two training frameworks are automatically entered to perform strategy optimization calculations, and the first training strategy and the second training strategy can be determined. Taking the first training framework as a training framework that supports fine-tuning training (such as LoRA training) and the second training framework as a training framework that supports full-parameter training as an example, in the first training framework, Liger operator optimization is used by default, and activation recalculation is turned on. The total available video memory is calculated based on the number of graphics cards and the size of the video memory in the hardware parameter information, and then the total training video memory corresponding to each candidate training strategy is determined in turn according to the optimization order of multiple candidate training strategies in the first training framework. If the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, the current candidate training strategy can be used as the first training strategy. For example, the first training strategy can be zero3; in the second training framework, SP is turned on by default, and recalculation is set to None. The total available video memory is calculated based on the number of graphics cards and the size of the video memory in the hardware parameter information, and then the pipeline parallelism and tensor parallelism of the second training framework are configured based on the preset configuration information. For example, PP=1 is set first, and TP increases from 1 to 4 in sequence. It stops when the total training video memory ≤ the total available video memory is satisfied. For example, the second training strategy can be PP=1, TP=4.

[0062] Step S203, determining a target training strategy corresponding to the model to be trained from the first training strategy and the second training strategy according to a preset number of training steps and a preset data set.

[0063] Optionally, the training strategies determined by the two training frameworks are automatically verified to determine a better target training strategy, that is, according to a preset number of training steps and a preset data set, the target training strategy corresponding to the model to be trained is determined from the first training strategy and the second training strategy. For example, according to the first training strategy and the second training strategy, the model is trained for 10 steps using a preset data set for strategy verification, and the training speeds (such as throughput) of the two training strategies are compared to determine a better target training strategy.

[0064] Optionally, after determining the target training strategy, the target training strategy can be fed back to the user. The user does not need to perceive the underlying specific training framework, but only needs to input or provide automatic reading of hardware parameters, model parameters and training parameters to automatically run the calculation to obtain a better training strategy.

[0065] In this solution, it is possible to automatically determine a better training strategy based on the actual hardware configuration, model parameters, etc., and automatically verify the strategy based on the preset number of training steps and the preset data set, thereby improving the accuracy of the determined strategy, effectively saving time and cost, and achieving the purpose of increasing the training speed of large models, thereby achieving the technical effect of improving the accuracy and efficiency of training strategy selection, and further solving the technical problem in the related technology of determining the training strategy of the model based on expert experience, resulting in low accuracy of the determined training strategy.

[0066] In order to determine the first training strategy, in the method for determining the large model training strategy provided in Example 1 of the present application, the first training framework is a training framework that supports fine-tuning training, and the first training framework is used to determine the first training strategy based on hardware parameter information, model parameter information and training parameter information, including: configuring the operator optimization and recalculation of the first training framework to be turned on; determining the total available video memory based on the hardware parameter information; obtaining the arrangement order of multiple candidate training strategies in the first training framework, and based on the arrangement order, determining the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies based on the model parameter information and the training parameter information; determining the first training strategy from multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies.

[0067] Optionally, taking the first training framework as a training framework that supports fine-tuning training (such as LoRA training) as an example, the operator optimization and recalculation of the first training framework are configured to be turned on, for example, Liger operator optimization is used by default, and activation recalculation is turned on. Then, the total available video memory is determined based on the hardware parameter information, and the formula is as follows:

[0068] Total available video memory = number of graphics cards * video memory size

[0069] Optionally, the multiple candidate training strategies in the first training framework include: zero1, zero2, zero3, zero3 and offload parameter, zero3 and offload optimizer, zero3 and offload parameter and optimizer and gradient (offload all). According to the training speed from fast to slow and the memory occupied size from large to small, the arrangement order is as follows: zero1>zero2>zero3>zero3+offload parameter>zero3+offloadoptimizer>zero3+offload all.

[0070] Optionally, based on the arrangement order, the total training video memory corresponding to each candidate training strategy (i.e., the video memory required for model training) can be determined according to the model parameter information and the training parameter information. Among them, the total training video memory size can be estimated according to the model parameter information and the training parameter information through the currently widely used calculation formula, or the total training video memory size can be obtained by training the model for one step, or the total training video memory size can be obtained by combining formula calculation and model training. For example, when the calculated total training video memory is greater than the total available video memory, there is no need to perform model training, and the total training video memory calculation of the next candidate training strategy is directly performed; when the calculated total training video memory is less than or equal to the total available video memory, the number of training steps is set to 1 step, and model training is performed to obtain the total training video memory. At this time, if the total training video memory obtained by training is still less than or equal to the total available video memory, the subsequent steps can be performed.

[0071] Optionally, the first training strategy can be determined from multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to each candidate training strategy. For example, the total available video memory and the total training video memory corresponding to the current candidate training strategy can be compared, and if the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, the current candidate training strategy can be used as the first training strategy.

[0072] It should be noted that, by determining the total available video memory and the total training video memory, the first training strategy is accurately determined.

[0073] In order to accurately determine the first training strategy, in the large model training strategy determination method provided in the first embodiment of the present application, the first training strategy is determined from multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies in the multiple candidate training strategies, including: if the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, the configuration recalculation is set to the closed state, and the total training video memory corresponding to the current candidate training strategy is re-determined based on the model parameter information and the training parameter information; if the total training video memory corresponding to the re-determined current candidate training strategy is less than or equal to the total available video memory, the current candidate training strategy is used as the first training strategy.

[0074] Optionally, since recalculation is enabled by default, and recalculation itself will increase the calculation time, it is necessary to minimize the use of recalculation while satisfying the memory constraint. For example, if the total training memory of the current candidate training strategy is still less than or equal to the total available memory when recalculation is disabled, then there is no need to recalculate, and directly adopting the current strategy can increase the training speed. Therefore, if the total training memory corresponding to the current candidate training strategy is less than or equal to the total available memory, for example, when zero2 or zero3 already satisfies the total training memory ≤ total available memory, the key step memory and recalculation logic are optimized, that is, the recalculation is configured to be disabled, and the total training memory corresponding to the current candidate training strategy is re-determined based on the model parameter information and the training parameter information. If the total training memory corresponding to the re-determined current candidate training strategy is less than or equal to the total available memory, the current candidate training strategy is used as the first training strategy. That is, the goal is to satisfy the requirement that the total training video memory is less than or equal to the total available video memory when recalculation is turned off, and then try to turn off recalculation. For example, if the current single-card batch size (batch size of data on a single graphics card) bs0>1, try to turn off activation recalculation and gradually increase the single-card batch size from 1 to bs0 (for example, the input batch size bs0=4) until the video memory overflows (Out Of Memory, referred to as OOM) and the current strategy is output. When the single-card batch size (perdevice batch size, referred to as pbs) changes, the total batch size (the total batch size of the training data, global batch size, referred to as gbs) is kept unchanged by automatically setting the gradient accumulation value (ga) to ensure the consistency of the training effect. For example, when the number of graphics cards is 8, pbs=1, ga=16, gbs=8*1*16=128, when pbs increases to 2, ga=8, gbs=8*2*8=128 remain unchanged.

[0075] It should be noted that by optimizing the key step video memory and recalculation logic, that is, by introducing the Liger operator optimization to reduce video memory and speed up, and reducing the batch size of a single card to reduce video memory (while increasing the number of gradient accumulation to keep the total batch size unchanged to ensure the training effect), it is achieved that the total training video memory can be less than or equal to the total available video memory when recalculation is turned off, that is, recalculation can be turned off to speed up.

[0076] In order to determine the first training strategy, in the method for determining the large model training strategy provided in Example 1 of the present application, determining the first training strategy from multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies includes: if the total training video memory corresponding to the current candidate training strategy is greater than the total available video memory, then based on the arrangement order, determining the total training video memory corresponding to the next candidate training strategy of the current candidate training strategy based on the model parameter information and the training parameter information, until the total training video memory corresponding to the candidate training strategy is less than or equal to the total available video memory, so as to obtain the first training strategy.

[0077] Optionally, after calculating the total training video memory of the current training strategy in accordance with the aforementioned arrangement order, the size of the total training video memory is compared with the total available video memory. If the total training video memory of the current training strategy is ≤ the total available video memory, there is no need to calculate the total training video memory of the subsequent training strategy. Otherwise, the judgment is continued in sequence until the total training video memory ≤ the total available video memory is satisfied, and the current strategy is output. That is, if the total training video memory corresponding to the current candidate training strategy is greater than the total available video memory, the total training video memory corresponding to the next candidate training strategy of the current candidate training strategy is determined based on the arrangement order according to the model parameter information and the training parameter information, until the total training video memory corresponding to the candidate training strategy is less than or equal to the total available video memory, and the first training strategy is obtained.

[0078] It should be noted that, based on the arrangement order, by comparing the sizes of the total training video memory and the total available video memory, the efficiency of determining the first training strategy is improved.

[0079] In order to determine the second training strategy, in the method for determining the large model training strategy provided in Example 1 of the present application, the second training framework is a training framework that supports full-parameter training, and the second training framework is used to determine the second training strategy based on hardware parameter information, model parameter information and training parameter information, including: configuring the sequence parallelism of the second training framework to be on, and configuring the recalculation to be off; determining the total available video memory based on the hardware parameter information; configuring the pipeline parallelism, tensor parallelism and data parallelism of the second training framework based on preset configuration information to obtain multiple candidate training strategies for the second training framework; determining the total training video memory corresponding to the candidate training strategy among the multiple candidate training strategies based on the model parameter information and the training parameter information; determining the second training strategy from the multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategy among the multiple candidate training strategies.

[0080] Optionally, taking the second training framework as an example, the sequence parallelism of the second training framework is configured to be turned on, and the recalculation is configured to be turned off, for example, SP is turned on by default, and the recalculation is set to None. Then, the total available video memory is determined based on the hardware parameter information, and the formula is as follows:

[0081] Total available video memory = number of graphics cards * video memory size

[0082] Optionally, based on the preset configuration information, the pipeline parallelism, tensor parallelism and data parallelism of the second training framework are configured to obtain multiple candidate training strategies of the second training framework. For example, the preset configuration information includes: TP values ​​increase in sequence of 1, 2, 4, and 8, PP values ​​increase in sequence of 1, 2, 4, and 8, and the number of graphics cards is a positive integer multiple of TP*PP.

[0083] Optionally, the total training video memory corresponding to each candidate training strategy can be determined based on the model parameter information and the training parameter information, and then the second training strategy can be determined from multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to each candidate training strategy. Among them, the total training video memory size can be estimated based on the model parameter information and the training parameter information through the currently widely used calculation formula, or the total training video memory size can be obtained by training the model for 1 step, or the total training video memory size can be obtained by combining formula calculation and model training. For example, when the calculated total training video memory is greater than the total available video memory, there is no need to perform model training, and the total training video memory calculation of the next candidate training strategy is directly performed; when the calculated total training video memory is less than or equal to the total available video memory, the number of training steps is set to 1, and model training is performed to obtain the total training video memory. At this time, if the total training video memory obtained by training is still less than or equal to the total available video memory, the subsequent steps can be performed.

[0084] For example, the total available video memory and the total training video memory corresponding to the current candidate training strategy may be compared. If the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, the current candidate training strategy may be used as the second training strategy.

[0085] It should be noted that, by determining the total available video memory and the total training video memory, the second training strategy is accurately determined.

[0086] In order to accurately determine the second training strategy, in the method for determining the large model training strategy provided in Example 1 of the present application, based on the preset configuration information, the pipeline parallelism, tensor parallelism and data parallelism of the second training framework are configured to obtain multiple candidate training strategies for the second training framework, including: configuring the pipeline parallelism to be a first preset value, and configuring the tensor parallelism to sequentially take values ​​according to the preset values ​​in the first preset value set, to obtain a first candidate training strategy set; configuring the tensor parallelism to be a second preset value, and configuring the pipeline parallelism to sequentially take values ​​according to the preset values ​​in the second preset value set, to obtain a second candidate training strategy set; configuring the tensor parallelism to be a third preset value, and configuring the pipeline parallelism to sequentially take values ​​according to the preset values ​​in the second preset value set, to obtain a third candidate training strategy set; configuring the tensor parallelism to be a third preset value, configuring the pipeline parallelism to be a third preset value, and configuring the recalculated states based on multiple preset states in sequence to obtain a fourth candidate training strategy set; determining multiple candidate training strategies for the second training framework based on the first candidate training strategy set, the second candidate training strategy set, the third candidate training strategy set and the fourth candidate training strategy set.

[0087] Optionally, the pipeline parallelism is configured to be a first preset value (for example, PP=1), and the tensor parallelism is configured to sequentially take values ​​according to the preset values ​​in the first preset value set (such as 1, 2, 4) (for example, TP increases from 1 to 4 in sequence), and a first candidate training strategy set is obtained. When the total training video memory ≤ the total available video memory is satisfied, the process stops, and the current strategy is output as the second training strategy. For example, the current strategy is: SP is turned on, recalculation is None, PP=1, TP=4.

[0088] Optionally, the tensor parallelism is configured to be a second preset value (for example, TP=4), and the pipeline parallelism is configured to sequentially take values ​​according to the preset values ​​in the second preset value set (such as 2, 4, 8) (for example, PP increases from 2 to 8 in sequence), and a second candidate training strategy set is obtained. When the total training video memory ≤ the total available video memory is satisfied, the process stops and the current strategy is output as the second training strategy.

[0089] Optionally, the tensor parallelism is configured to be a third preset value (for example, TP=8), and the pipeline parallelism is configured to sequentially take values ​​according to the preset values ​​in the second preset value set (for example, PP increases from 2 to 8 in sequence), to obtain a third candidate training strategy set. When the total training video memory ≤ the total available video memory is satisfied, stop, and output the current strategy as the second training strategy.

[0090] Optionally, configure the tensor parallelism to a third preset value, configure the pipeline parallelism to a third preset value, and configure the recalculation state in sequence based on multiple preset states (such as Selective, Full+uniform, Full+block, Full+uniform+distributed optimizer), to obtain a fourth candidate training strategy set. For example, when TP=8, PP=8, the total training video memory ≤ total available video memory is still not satisfied, then the recalculation state is configured in the following order: Selective>Full+uniform>Full+block>Full+uniform+distributed optimizer is turned on. Among them, when the total training video memory ≤ total available video memory is satisfied, stop and output the current strategy as the second training strategy.

[0091] Optionally, the first candidate training strategy set, the second candidate training strategy set, the third candidate training strategy set and the fourth candidate training strategy set are used as multiple candidate training strategies of the second training framework.

[0092] Optionally, the context parallelism CP defaults to 1. When the sequence length exceeds 64k, CP is set to 2 and 4 respectively. The value of data parallelism DP is calculated by the following formula:

[0093]

[0094] It should be noted that by configuring the pipeline parallelism, tensor parallelism and data parallelism of the second training framework, multiple parallel training strategies are determined, so that the better parallel strategy can be accurately determined.

[0095] In order to determine the second training strategy, in the method for determining the large model training strategy provided in Example 1 of the present application, the second training strategy is determined from multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies, including: if the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, then the current candidate training strategy is used as the second training strategy.

[0096] Optionally, during the configuration process, the total available video memory and the total training video memory corresponding to the current candidate training strategy are compared. If the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, the current candidate training strategy is used as the second training strategy, i.e., no subsequent strategy configuration is required.

[0097] It should be noted that by comparing the total available video memory with the total training video memory corresponding to the current candidate training strategy, the second training strategy is accurately determined.

[0098] In order to determine a better target training strategy, in the method for determining the large model training strategy provided in Example 1 of the present application, based on a preset number of training steps and a preset data set, determining the target training strategy corresponding to the model to be trained from the first training strategy and the second training strategy includes: performing model training on the model to be trained according to the first training strategy based on the preset number of training steps and the preset data set to obtain the throughput corresponding to the first training strategy; performing model training on the model to be trained according to the second training strategy based on the preset number of training steps and the preset data set to obtain the throughput corresponding to the second training strategy; and taking the training strategy with a large throughput as the target training strategy.

[0099] Optionally, the training strategies respectively determined by the two training frameworks are automatically verified, that is, according to the preset number of training steps and the preset data set, the model to be trained is trained according to the first training strategy to obtain the throughput corresponding to the first training strategy, and according to the preset number of training steps and the preset data set, the model to be trained is trained according to the second training strategy to obtain the throughput corresponding to the second training strategy, and then the training strategy with a large throughput is used as the target training strategy. For example, according to the first training strategy and the second training strategy, the model is trained for 10 steps using the preset data set for strategy verification, and the throughputs of the two training strategies are compared, and the training strategy with a large throughput is used as the target training strategy.

[0100] It should be noted that by calculating the throughput and taking the training strategy with a high throughput as the target training strategy, the model can process more data in a shorter time, thereby speeding up the training process and saving computing resources and time costs.

[0101] In an optional embodiment, it is possible to use Figure 3 The schematic diagram shown in FIG. 1 is used to determine the target training strategy. Figure 3 is a schematic diagram of an optional process for determining a target training strategy according to the first embodiment of the present application, such as Figure 3 As shown, two training frameworks are used to select strategies based on hardware parameter information, model parameter information and training parameter information, and each determines a training strategy. Then, data, models and preset steps are used to perform model training (i.e., automatic operation verification) according to the determined training strategy, and a better target training strategy can be determined. The specific implementation details refer to the aforementioned embodiment and will not be repeated here.

[0102] In an embodiment of the present application, by obtaining hardware parameter information, model parameter information and training parameter information corresponding to the model to be trained; using a first training framework to determine a first training strategy based on the hardware parameter information, model parameter information and training parameter information, and using a second training framework to determine a second training strategy based on the hardware parameter information, model parameter information and training parameter information, wherein the training algorithms corresponding to the first training framework and the second training framework are different; according to a preset number of training steps and a preset data set, a target training strategy corresponding to the model to be trained is determined from the first training strategy and the second training strategy, thereby realizing automatic determination of a better training strategy based on actual hardware configuration, model parameters, etc., and automatic strategy verification based on a preset number of training steps and a preset data set, thereby improving the accuracy of the determined strategy, effectively saving time and cost, and achieving the purpose of improving the training speed of large models, thereby achieving the technical effect of improving the accuracy and efficiency of training strategy selection, and then solving the technical problem of low accuracy of the determined training strategy in the related technology that the training strategy of the model is determined based on expert experience.

[0103] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0104] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0105] Example 2

[0106] According to an embodiment of the present application, a method for generating a target model is also provided, such as Figure 4 As shown, the method includes:

[0107] Step S401, obtaining training data;

[0108] Step S402, using the target training strategy and training data to train the large model to be trained, to obtain a trained target large model, wherein the target training strategy is obtained by using any of the above methods for determining the large model training strategy.

[0109] Optionally, the large model to be trained is trained by determining a better target training strategy and training data to obtain a trained target large model, which effectively improves the training speed of the model, realizes the acceleration of large model training with adaptive parallel strategy and re-computation optimization, realizes the automatic determination of a better training strategy based on actual hardware configuration, model parameters, etc., and automatically verifies the strategy based on a preset number of training steps and a preset data set, thereby improving the accuracy of the determined strategy, effectively saving time and cost, and achieving the purpose of improving the training speed of the large model, thereby achieving the technical effect of improving the accuracy and efficiency of the selection of training strategies, and further solves the technical problem of low accuracy of the determined training strategy in the related technology based on expert experience to determine the training strategy of the model.

[0110] Example 3

[0111] According to an embodiment of the present application, a method for determining a large model training strategy is also provided, such as Figure 5 As shown, the method includes:

[0112] Step S501, obtaining hardware parameter information, model parameter information and training parameter information corresponding to the model to be trained uploaded by the client;

[0113] Step S502, in the cloud server, a first training framework is used to determine a first training strategy based on hardware parameter information, model parameter information, and training parameter information, and a second training framework is used to determine a second training strategy based on hardware parameter information, model parameter information, and training parameter information, wherein the first training framework and the second training framework have different training algorithms; according to a preset number of training steps and a preset data set, a target training strategy corresponding to the model to be trained is determined from the first training strategy and the second training strategy;

[0114] Step S503: Feedback the target training strategy to the client.

[0115] Through the above scheme, it is possible to automatically determine a better training strategy based on the actual hardware configuration, model parameters, etc., and automatically verify the strategy based on the preset training steps and preset data sets, thereby improving the accuracy of the determined strategy, effectively saving time and cost, and achieving the purpose of increasing the training speed of large models, thereby achieving the technical effect of improving the accuracy and efficiency of training strategy selection, and further solving the technical problem in the related technology of determining the training strategy of the model based on expert experience, resulting in low accuracy of the determined training strategy.

[0116] In the cloud server, the specific method for determining the large model training strategy is the same as the method in Example 1 and will not be repeated here.

[0117] It should be noted that, for the aforementioned method embodiments, for the sake of simplicity, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the present application is not limited by the described order of actions, because according to the present application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present application.

[0118] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods of each embodiment of the present application.

[0119] Example 4

[0120] According to an embodiment of the present application, a device for determining a large model training strategy for implementing the above-mentioned method for determining a large model training strategy is also provided, such as Figure 6 As shown, the device includes: a first acquisition unit 601, a first determination unit 602, and a second determination unit 603.

[0121] The first acquisition unit 601 is used to acquire hardware parameter information, model parameter information and training parameter information corresponding to the model to be trained;

[0122] A first determining unit 602 is used to determine a first training strategy based on hardware parameter information, model parameter information, and training parameter information using a first training framework, and to determine a second training strategy based on hardware parameter information, model parameter information, and training parameter information using a second training framework, wherein the first training framework and the second training framework have different training algorithms;

[0123] The second determining unit 603 is used to determine a target training strategy corresponding to the to-be-trained model from the first training strategy and the second training strategy according to a preset number of training steps and a preset data set.

[0124] In the device for determining the training strategy of a large model provided in the fourth embodiment of the present application, the hardware parameter information, model parameter information and training parameter information corresponding to the model to be trained are obtained by the first acquisition unit 601; the first determination unit 602 adopts the first training framework to determine the first training strategy based on the hardware parameter information, model parameter information and training parameter information, and adopts the second training framework to determine the second training strategy based on the hardware parameter information, model parameter information and training parameter information, wherein the training algorithms corresponding to the first training framework and the second training framework are different; the second determination unit 603 determines the target training strategy corresponding to the model to be trained from the first training strategy and the second training strategy according to the preset training steps and the preset data set. In this scheme, it is realized that a better training strategy is automatically determined according to the actual hardware configuration, model parameters, etc., and the strategy verification is automatically performed according to the preset training steps and the preset data set, thereby improving the accuracy of the determined strategy, effectively saving time and cost, and achieving the purpose of improving the training speed of the large model, thereby achieving the technical effect of improving the accuracy and efficiency of the selection of the training strategy, and then solving the technical problem that the accuracy of the determined training strategy is low in the related technology based on the expert experience to determine the training strategy of the model.

[0125] Optionally, in the large model training strategy determination device provided in Example 4 of the present application, the first determination unit 602 includes: a first configuration subunit, used to configure the operator optimization and recalculation of the first training framework to be in an on state; a first determination subunit, used to determine the total available video memory based on the hardware parameter information; a second determination subunit, used to obtain the arrangement order of multiple candidate training strategies in the first training framework, and based on the arrangement order, determine the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies according to the model parameter information and the training parameter information; a third determination subunit, used to determine the first training strategy from multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies.

[0126] Optionally, in the large model training strategy determination device provided in Example 4 of the present application, the third determination subunit includes: a first determination module, which is used to configure the recalculation to be closed if the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, and re-determine the total training video memory corresponding to the current candidate training strategy based on the model parameter information and the training parameter information; a second determination module, which is used to use the current candidate training strategy as the first training strategy if the total training video memory corresponding to the re-determined current candidate training strategy is less than or equal to the total available video memory.

[0127] Optionally, in the device for determining the large model training strategy provided in Example 4 of the present application, the third determination subunit includes: a third determination module, which is used to determine the total training video memory corresponding to the next candidate training strategy of the current candidate training strategy based on the arrangement order, according to the model parameter information and the training parameter information, if the total training video memory corresponding to the current candidate training strategy is greater than the total available video memory, until the total training video memory corresponding to the candidate training strategy is less than or equal to the total available video memory, to obtain the first training strategy.

[0128] Optionally, in the device for determining the large model training strategy provided in Example 4 of the present application, the first determination unit 602 also includes: a second configuration subunit, used to configure the sequence parallelism of the second training framework to be in an on state, and to configure the recalculation to be in an off state; a fourth determination subunit, used to determine the total available video memory based on the hardware parameter information; a third configuration subunit, used to configure the pipeline parallelism, tensor parallelism and data parallelism of the second training framework based on preset configuration information, to obtain multiple candidate training strategies for the second training framework; a fifth determination subunit, used to determine the total training video memory corresponding to the candidate training strategy among the multiple candidate training strategies based on the model parameter information and the training parameter information; a sixth determination subunit, used to determine the second training strategy from multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategy among the multiple candidate training strategies.

[0129] Optionally, in the device for determining the large model training strategy provided in Example 4 of the present application, the third configuration subunit includes: a first configuration module, used to configure pipeline parallelism to a first preset value, and configure tensor parallelism to sequentially take values ​​according to the preset values ​​in the first preset value set to obtain a first candidate training strategy set; a second configuration module, used to configure tensor parallelism to a second preset value, and configure pipeline parallelism to sequentially take values ​​according to the preset values ​​in the second preset value set to obtain a second candidate training strategy set; a third configuration module, used to configure tensor parallelism to a third preset value, and configure pipeline parallelism to sequentially take values ​​according to the preset values ​​in the second preset value set to obtain a third candidate training strategy set; a fourth configuration module, used to configure tensor parallelism to a third preset value, configure pipeline parallelism to a third preset value, and sequentially configure the recalculated state based on multiple preset states to obtain a fourth candidate training strategy set; a fourth determination module, used to determine multiple candidate training strategies for the second training framework based on the first candidate training strategy set, the second candidate training strategy set, the third candidate training strategy set and the fourth candidate training strategy set.

[0130] Optionally, in the device for determining the large model training strategy provided in Example 4 of the present application, the sixth determination subunit includes: a fifth determination module, which is used to use the current candidate training strategy as the second training strategy if the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory.

[0131] Optionally, in the device for determining the large model training strategy provided in Example 4 of the present application, the second determination unit 603 includes: a first processing subunit, used to perform model training on the training model according to a first training strategy based on a preset number of training steps and a preset data set, so as to obtain a throughput corresponding to the first training strategy; a second processing subunit, used to perform model training on the training model according to a second training strategy based on a preset number of training steps and a preset data set, so as to obtain a throughput corresponding to the second training strategy; and a seventh determination subunit, used to use the training strategy with a high throughput as the target training strategy.

[0132] It should be noted that the first acquisition unit 601, the first determination unit 602, and the second determination unit 603 described above correspond to steps S201 to S203 in Embodiment 1, and the examples and application scenarios implemented by the above units and corresponding steps are the same, but are not limited to the contents disclosed in the above Embodiment 1. It should be noted that the above modules, as part of the device, can be run in the computer terminal 10 provided in Embodiment 1.

[0133] It should be noted that the preferred implementation scheme involved in the above embodiments of the present application is the same as the scheme provided in Example 1, as well as the application scenario and implementation process, but is not limited to the scheme provided in Example 1.

[0134] Example 5

[0135] According to an embodiment of the present application, a device for generating a target model for implementing the above-mentioned method for generating a target model is also provided. Figure 7 As shown, the device includes: a second acquisition unit 701 and a training unit 702.

[0136] A second acquisition unit 701 is used to acquire training data;

[0137] The training unit 702 is used to train the large model to be trained using the target training strategy and training data to obtain a trained target large model, wherein the target training strategy is obtained by using any of the above-mentioned methods for determining the large model training strategy.

[0138] It should be noted here that the above-mentioned second acquisition unit 701 and training unit 702 correspond to steps S401 to S402 in Example 2. The instances and application scenarios implemented by the above-mentioned units and corresponding steps are the same, but are not limited to the contents disclosed in the above-mentioned Example 2.

[0139] Example 6

[0140] The embodiment of the present application may provide an electronic device, which may be any electronic device in a group of electronic devices. Optionally, in this embodiment, the electronic device may also be replaced by a terminal device such as a mobile terminal.

[0141] Optionally, in this embodiment, the electronic device may be located in at least one network device among a plurality of network devices of a computer network.

[0142] In this embodiment, the above-mentioned electronic device can execute the program code of the following steps in the method for determining the large model training strategy: obtaining hardware parameter information, model parameter information and training parameter information corresponding to the model to be trained; using a first training framework to determine the first training strategy based on the hardware parameter information, model parameter information and training parameter information, and using a second training framework to determine the second training strategy based on the hardware parameter information, model parameter information and training parameter information, wherein the training algorithms corresponding to the first training framework and the second training framework are different; based on a preset number of training steps and a preset data set, determining the target training strategy corresponding to the model to be trained from the first training strategy and the second training strategy.

[0143] The above-mentioned electronic device can also execute the program code of the following steps in the method for determining the large model training strategy: configure the operator optimization and recalculation of the first training framework to be turned on; determine the total available video memory based on the hardware parameter information; obtain the arrangement order of multiple candidate training strategies in the first training framework, and based on the arrangement order, determine the total training video memory corresponding to the candidate training strategies in the multiple candidate training strategies according to the model parameter information and the training parameter information; determine the first training strategy from the multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies in the multiple candidate training strategies.

[0144] The above-mentioned electronic device can also execute the program code of the following steps in the method for determining the large model training strategy: if the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, the configuration recalculation is set to the closed state, and the total training video memory corresponding to the current candidate training strategy is re-determined based on the model parameter information and the training parameter information; if the total training video memory corresponding to the re-determined current candidate training strategy is less than or equal to the total available video memory, the current candidate training strategy is used as the first training strategy.

[0145] The above-mentioned electronic device can also execute the program code of the following steps in the method for determining the large model training strategy: if the total training video memory corresponding to the current candidate training strategy is greater than the total available video memory, then based on the arrangement order, determine the total training video memory corresponding to the next candidate training strategy of the current candidate training strategy according to the model parameter information and the training parameter information, until the total training video memory corresponding to the candidate training strategy is less than or equal to the total available video memory, and obtain the first training strategy.

[0146] The above-mentioned electronic device can also execute the program code of the following steps in the method for determining the large model training strategy: configure the sequence parallelism of the second training framework to be on, and configure the recalculation to be off; determine the total available video memory based on the hardware parameter information; based on the preset configuration information, configure the pipeline parallelism, tensor parallelism and data parallelism of the second training framework to obtain multiple candidate training strategies for the second training framework; determine the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies based on the model parameter information and the training parameter information; determine the second training strategy from the multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies.

[0147] The electronic device can also execute the program code of the following steps in the method for determining the large model training strategy: configure the pipeline parallelism to the first preset value, and configure the tensor parallelism to sequentially take values ​​according to the preset values ​​in the first preset value set to obtain the first candidate training strategy set; configure the tensor parallelism to the second preset value, and configure the pipeline parallelism to sequentially take values ​​according to the preset values ​​in the second preset value set to obtain the second candidate training strategy set; configure the tensor parallelism to the third preset value, and configure the pipeline parallelism to sequentially take values ​​according to the preset values ​​in the second preset value set to obtain the third candidate training strategy set; configure the tensor parallelism to the third preset value, configure the pipeline parallelism to the third preset value, and configure the recalculated states in sequence based on multiple preset states to obtain the fourth candidate training strategy set; determine multiple candidate training strategies for the second training framework based on the first candidate training strategy set, the second candidate training strategy set, the third candidate training strategy set and the fourth candidate training strategy set.

[0148] The electronic device may also execute the following program codes in the method for determining the large model training strategy: if the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, the current candidate training strategy is used as the second training strategy.

[0149] The above-mentioned electronic device can also execute the program code of the following steps in the method for determining the large model training strategy: based on a preset number of training steps and a preset data set, model training is performed on the training model according to a first training strategy to obtain the throughput corresponding to the first training strategy; based on a preset number of training steps and a preset data set, model training is performed on the training model according to a second training strategy to obtain the throughput corresponding to the second training strategy; and the training strategy with a large throughput is used as the target training strategy.

[0150] Optionally, Figure 8 is a structural block diagram of an electronic device according to an embodiment of the present application. Figure 8 As shown, the electronic device 80 may include: one or more ( Figure 8Only one is shown in the figure) processor 802 and memory 804. The electronic device 80 may also include a storage controller, through which the memory 804 is controlled and managed; the electronic device 80 may also include a peripheral interface, through which the radio frequency module, audio module and display screen are connected.

[0151] Among them, the memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the method and device for determining the large model training strategy in the embodiment of the present application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, that is, the method for determining the large model training strategy described above is realized. The memory may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include a memory remotely arranged relative to the processor, and these remote memories may be connected to the terminal 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0152] The processor can call the information and application programs stored in the memory through the transmission device to perform the following steps: obtain hardware parameter information, model parameter information and training parameter information corresponding to the model to be trained; use the first training framework to determine the first training strategy based on the hardware parameter information, model parameter information and training parameter information, and use the second training framework to determine the second training strategy based on the hardware parameter information, model parameter information and training parameter information, wherein the training algorithms corresponding to the first training framework and the second training framework are different; determine the target training strategy corresponding to the model to be trained from the first training strategy and the second training strategy based on the preset number of training steps and the preset data set.

[0153] Optionally, the processor may also execute program code of the following steps: configuring operator optimization and recalculation of the first training framework to be turned on; determining the total available video memory based on the hardware parameter information; obtaining the arrangement order of multiple candidate training strategies in the first training framework, and based on the arrangement order, determining the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies based on the model parameter information and the training parameter information; determining the first training strategy from the multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies.

[0154] Optionally, the processor may also execute the following program code: if the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, the configuration recalculation is set to a closed state, and the total training video memory corresponding to the current candidate training strategy is re-determined based on the model parameter information and the training parameter information; if the total training video memory corresponding to the re-determined current candidate training strategy is less than or equal to the total available video memory, the current candidate training strategy is used as the first training strategy.

[0155] Optionally, the processor may also execute the following program code: if the total training video memory corresponding to the current candidate training strategy is greater than the total available video memory, then based on the arrangement order, determine the total training video memory corresponding to the next candidate training strategy of the current candidate training strategy according to the model parameter information and the training parameter information, until the total training video memory corresponding to the candidate training strategy is less than or equal to the total available video memory, thereby obtaining the first training strategy.

[0156] Optionally, the processor may also execute program code of the following steps: configuring the sequence parallelism of the second training framework to be on, and configuring the recalculation to be off; determining the total available video memory based on the hardware parameter information; configuring the pipeline parallelism, tensor parallelism, and data parallelism of the second training framework based on preset configuration information to obtain multiple candidate training strategies for the second training framework; determining the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies based on the model parameter information and the training parameter information; determining the second training strategy from the multiple candidate training strategies based on the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies.

[0157] Optionally, the processor may also execute the following steps of program code: configuring pipeline parallelism to a first preset value, and configuring tensor parallelism to sequentially take values ​​according to the preset values ​​in the first preset value set, to obtain a first candidate training strategy set; configuring tensor parallelism to a second preset value, and configuring pipeline parallelism to sequentially take values ​​according to the preset values ​​in the second preset value set, to obtain a second candidate training strategy set; configuring tensor parallelism to a third preset value, and configuring pipeline parallelism to sequentially take values ​​according to the preset values ​​in the second preset value set, to obtain a third candidate training strategy set; configuring tensor parallelism to a third preset value, configuring pipeline parallelism to a third preset value, and configuring the recalculated states based on multiple preset states in sequence to obtain a fourth candidate training strategy set; determining multiple candidate training strategies for the second training framework based on the first candidate training strategy set, the second candidate training strategy set, the third candidate training strategy set, and the fourth candidate training strategy set.

[0158] Optionally, the processor may further execute program code of the following steps: if the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, then use the current candidate training strategy as the second training strategy.

[0159] Optionally, the processor may also execute the program code of the following steps: performing model training on the training model according to a first training strategy based on a preset number of training steps and a preset data set to obtain a throughput corresponding to the first training strategy; performing model training on the training model according to a second training strategy based on a preset number of training steps and a preset data set to obtain a throughput corresponding to the second training strategy; and taking the training strategy with a high throughput as the target training strategy.

[0160] It can be understood by those skilled in the art that Figure 8 The structure shown is for illustration only, and the electronic device may also be a smart phone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (Mobile Internet Devices, MID), a PAD, or other terminal devices. Figure 8 The structure of the electronic device is not limited. Figure 8 More or fewer components (such as network interfaces, display devices, etc.) shown in, or having Figure 8 Different configurations are shown.

[0161] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing the hardware related to the terminal device through a program, and the program can be stored in a computer-readable storage medium, and the storage medium may include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0162] Example 7

[0163] The embodiment of the present application also provides a computer-readable storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the method for determining the large model training strategy provided in the first embodiment.

[0164] Optionally, in this embodiment, the above storage medium may be located in any electronic device in a group of electronic devices in a computer network, or in any mobile terminal in a group of mobile terminals.

[0165] Example 8

[0166] The embodiment of the present application also provides a computer program product. Optionally, in this embodiment, the computer program product may include a computer program, and when the computer program is executed by a processor, the method for determining the large model training strategy provided in the first embodiment is implemented.

[0167] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0168] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0169] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic, for example, the division of units is only a logical function division, and there may be other division methods in actual implementation, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0172] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or all or part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of each embodiment method of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0173] The above are only preferred implementations of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A method for determining a large model training strategy, characterized in that: include: Obtain hardware parameter information, model parameter information, and training parameter information corresponding to the model to be trained; A first training framework is used to determine a first training strategy based on the hardware parameter information, the model parameter information, and the training parameter information, and a second training framework is used to determine a second training strategy based on the hardware parameter information, the model parameter information, and the training parameter information, wherein the first training framework and the second training framework have different training algorithms. According to a preset number of training steps and a preset data set, a target training strategy corresponding to the model to be trained is determined from the first training strategy and the second training strategy.

2. The method according to claim 1, characterized in that The first training framework is a training framework that supports fine-tuning training, and using the first training framework to determine a first training strategy based on the hardware parameter information, the model parameter information, and the training parameter information includes: Configure the operator optimization and recalculation of the first training framework to be turned on; Determining the total available video memory according to the hardware parameter information; Obtaining an arrangement order of multiple candidate training strategies in the first training framework, and determining a total training video memory corresponding to a candidate training strategy among the multiple candidate training strategies based on the arrangement order, according to the model parameter information and the training parameter information; The first training strategy is determined from the multiple candidate training strategies according to the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies.

3. The method according to claim 2, characterized in that Determining the first training strategy from the multiple candidate training strategies according to the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies includes: If the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, configure the recalculation to be closed, and re-determine the total training video memory corresponding to the current candidate training strategy based on the model parameter information and the training parameter information; If the total training video memory corresponding to the re-determined current candidate training strategy is less than or equal to the total available video memory, the current candidate training strategy is used as the first training strategy.

4. The method according to claim 2, characterized in that: Determining the first training strategy from the multiple candidate training strategies according to the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies includes: If the total training video memory corresponding to the current candidate training strategy is greater than the total available video memory, then based on the arrangement order, the total training video memory corresponding to the next candidate training strategy of the current candidate training strategy is determined according to the model parameter information and the training parameter information, until the total training video memory corresponding to the candidate training strategy is less than or equal to the total available video memory, to obtain the first training strategy.

5. The method according to claim 1, characterized in that The second training framework is a training framework that supports full-parameter training, and using the second training framework to determine the second training strategy based on the hardware parameter information, the model parameter information, and the training parameter information includes: Configure the sequence and parallel behavior of the second training framework to be in an on state, and configure the recalculation to be in an off state; Determining the total available video memory according to the hardware parameter information; Based on the preset configuration information, pipeline parallelism, tensor parallelism, and data parallelism of the second training framework are configured to obtain multiple candidate training strategies of the second training framework; Determining a total training video memory corresponding to a candidate training strategy among the multiple candidate training strategies according to the model parameter information and the training parameter information; The second training strategy is determined from the multiple candidate training strategies according to the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies.

6. The method according to claim 5, characterized in that Based on the preset configuration information, pipeline parallelism, tensor parallelism, and data parallelism of the second training framework are configured to obtain multiple candidate training strategies for the second training framework, including: The pipeline parallelism is configured to be a first preset value, and the tensor parallelism is configured to sequentially take values ​​according to the preset values ​​in the first preset value set, to obtain a first candidate training strategy set; Configuring the tensor parallel to be a second preset value, and configuring the pipeline parallel to sequentially take values ​​according to the preset values ​​in the second preset value set, to obtain a second candidate training strategy set; Configuring the tensor parallel to be a third preset value, and configuring the pipeline parallel to sequentially take values ​​according to the preset values ​​in the second preset value set, to obtain a third candidate training strategy set; Configuring the tensor parallelism to be the third preset value, configuring the pipeline parallelism to be the third preset value, and configuring the recalculated states in sequence based on a plurality of preset states to obtain a fourth candidate training strategy set; A plurality of candidate training strategies of the second training framework are determined according to the first candidate training strategy set, the second candidate training strategy set, the third candidate training strategy set, and the fourth candidate training strategy set.

7. The method according to claim 5, characterized in that Determining the second training strategy from the multiple candidate training strategies according to the total available video memory and the total training video memory corresponding to the candidate training strategies among the multiple candidate training strategies includes: If the total training video memory corresponding to the current candidate training strategy is less than or equal to the total available video memory, the current candidate training strategy is used as the second training strategy.

8. The method according to claim 1, characterized in that Determining the target training strategy corresponding to the to-be-trained model from the first training strategy and the second training strategy according to a preset number of training steps and a preset data set includes: According to the preset number of training steps and the preset data set, model training is performed on the to-be-trained model according to the first training strategy to obtain a throughput rate corresponding to the first training strategy; According to the preset number of training steps and the preset data set, model training is performed on the model to be trained according to the second training strategy to obtain a throughput rate corresponding to the second training strategy; A training strategy with a large throughput is used as the target training strategy.

9. A method for generating a target model, characterized in that: include: Get training data; The large model to be trained is trained using the target training strategy and the training data to obtain a trained target large model, wherein the target training strategy is obtained using the method for determining the large model training strategy described in any one of claims 1 to 8.

10. A method for determining a large model training strategy, characterized in that: include: Obtain the hardware parameter information, model parameter information, and training parameter information corresponding to the model to be trained uploaded by the client; In the cloud server, a first training framework is used to determine a first training strategy based on the hardware parameter information, the model parameter information and the training parameter information, and a second training framework is used to determine a second training strategy based on the hardware parameter information, the model parameter information and the training parameter information, wherein the first training framework and the second training framework have different training algorithms; according to a preset number of training steps and a preset data set, a target training strategy corresponding to the model to be trained is determined from the first training strategy and the second training strategy; The target training strategy is fed back to the client.

11. A device for determining a large model training strategy, characterized in that: include: A first acquisition unit, used to acquire hardware parameter information, model parameter information and training parameter information corresponding to the model to be trained; A first determining unit is used to determine a first training strategy based on the hardware parameter information, the model parameter information and the training parameter information using a first training framework, and to determine a second training strategy based on the hardware parameter information, the model parameter information and the training parameter information using a second training framework, wherein the first training framework and the second training framework have different training algorithms; The second determination unit is used to determine the target training strategy corresponding to the model to be trained from the first training strategy and the second training strategy according to a preset number of training steps and a preset data set.

12. An electronic device, characterized in that: include: A memory storing an executable program; A processor, used to run the program, wherein when the program is running, the method for determining the large model training strategy of any one of claims 1 to 10 is executed.

13. A computer program product, characterized in that It comprises a computer program or instructions, which, when executed by a processor, implements a method for determining a large model training strategy according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the storage medium is located is controlled to execute the method for determining the large model training strategy of any one of claims 1 to 10.

Images