Model fusion method and apparatus, and electronic device

Abstract

A model fusion method and apparatus, and an electronic device are provided.SOLUTION: In a pre-training process of a large-scale model, a main process is called to buffer intermediate model parameters in the pre-training process into a main buffer, and a sub-process is called through the main process to read the intermediate model parameters in the main buffer and perform parameter fusion processing based on the intermediate model parameters. The main process and the sub-process may be performed asynchronously, so that the parameter fusion processing may be performed on the intermediate model parameters obtained by pre-training the large-scale model while pre-training the large-scale model, thereby improving the model fusion efficiency without affecting the pre-training efficiency of the large-scale model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to the technical field of artificial intelligence, in particular to technical fields such as deep learning, cloud computing, and large-scale models, and in particular to model fusion methods, apparatuses, and electronic devices. [Background technology]

[0002] Currently, in the large-scale model pre-training process, a fusion process can be performed on the model parameters obtained by training in each training cycle in the large-scale model pre-training process to improve the performance of the pre-trained large-scale model.

[0003] Among these, the fusion method is mainly online fusion, which involves the pre-training process of a large-scale model and the fusion process of intermediate model parameters obtained through training, thereby reducing the efficiency of pre-training a large-scale model. Summary of the Invention

[0004] The present disclosure provides a model fusion method, apparatus, and electronic device.

[0005] According to one aspect of the present disclosure, there is provided a model fusion method, the model fusion method including the steps of: in a pre-training process of a large-scale model, buffering intermediate model parameters of the pre-training process in a main buffer by invoking a main process; and reading the intermediate model parameters in the main buffer by invoking a sub-process via the main process, and performing a parameter fusion process based on the intermediate model parameters.

[0006] According to another aspect of the present disclosure, there is provided a model fusion device including: a first calling module configured to buffer intermediate model parameters of a pre-training process of a large-scale model in a main buffer by calling a main process; and a second calling module configured to read the intermediate model parameters in the main buffer by calling a sub-process via the main process and perform parameter fusion processing based on the intermediate model parameters.

[0007] According to another aspect of the present disclosure, there is provided an electronic device including at least one processor and a memory communicatively connected to the at least one processor, wherein the memory stores instructions executable by the at least one processor, and wherein execution of the instructions by the at least one processor enables the at least one processor to perform the model fusion method of the present disclosure.

[0008] According to another aspect of the present disclosure, there is provided a non-transitory computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to cause a computer to perform the model fusion method of the present disclosure.

[0009] According to another aspect of the present disclosure, there is provided a computer program, which, when executed by a processor, implements the steps of the model fusion method of the present disclosure.

[0010] It should be understood that the contents of this Summary are not intended to identify key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will be readily apparent from the following description. [Brief explanation of the drawings]

[0011] The drawings are used for a better understanding of the present invention and are not intended to limit the present disclosure. [Figure 1] FIG. 1 is a schematic diagram according to a first embodiment of the present disclosure. [Figure 2] FIG. 10 is a schematic diagram according to a second embodiment of the present disclosure. [Figure 3] FIG. 10 is a schematic diagram according to a third embodiment of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram of a framework of main processes and sub-processes. [Figure 5] FIG. 1 is a schematic diagram of the execution of a main process and sub-processes. [Figure 6] FIG. 10 is a schematic diagram according to a fourth embodiment of the present disclosure. [Figure 7] FIG. 1 is a block diagram of an electronic device for implementing the model fusion method of an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, exemplary embodiments of the present disclosure will be described with reference to the drawings. For ease of understanding, various details of the embodiments of the present disclosure are included therein and should be considered as merely illustrative. Therefore, those skilled in the art should recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for clarity and conciseness, the following description will omit descriptions of well-known functions and structures.

[0013] Currently, in the large-scale model pre-training process, a fusion process can be performed on the model parameters obtained by training in each training cycle in the large-scale model pre-training process to improve the performance of the pre-trained large-scale model.

[0014] Among these, the fusion method is mainly online fusion, which involves the pre-training process of a large-scale model and the fusion process of intermediate model parameters obtained through training, thereby reducing the efficiency of pre-training a large-scale model.

[0015] In response to the above problems, the present disclosure provides a model fusion method, apparatus, and electronic device. 1 is a schematic diagram of a first embodiment of the present disclosure. The model fusion method of the embodiment of the present disclosure can be applied to a model fusion apparatus, which can be configured in an electronic device, thereby enabling the electronic device to perform the model fusion function.

[0016] The electronic device may be any device with computing capabilities, such as a personal computer (PC), a mobile terminal, a server, a cluster, etc. The mobile terminal may be a hardware device with a variety of operating systems, touchscreens, and / or display screens, such as an in-vehicle device, a mobile phone, a tablet computer, a personal digital assistant, a wearable device, a smart speaker, a server, a server cluster, etc.

[0017] The model fusion apparatus may be software in an electronic device, such as model fusion software, etc. In the following embodiment, an example will be described in which the execution subject is an electronic device.

[0018] As shown in FIG. 1, the model fusion method may include the following steps 101 and 102. In step 101, in the large-scale model pre-training process, the main process is invoked to buffer the intermediate model parameters of the pre-training process into the main buffer.

[0019] In an embodiment of the present disclosure, for example, the number of main processes is one, and the intermediate model parameters stored in the main buffer corresponding to the main process are all model parameters in the large-scale model. Correspondingly, the number of sub-processes is one. The main process and the sub-processes have a caller-callee relationship. That is, the main process can call the sub-processes, and the sub-processes can be called by the main process.

[0020] In another example, in the embodiment of the present disclosure, the number of main processes may be multiple. The intermediate model parameters stored in the main buffer corresponding to each main process may be local model parameters within the large-scale model. Correspondingly, the number of sub-processes may be multiple, and the main process and the sub-process have a one-to-one correspondence. The main process and the corresponding sub-process have a caller-callee relationship.

[0021] The main buffers corresponding to multiple main processes can be located in different Graphics Processing Units (GPUs), so the pre-training process of a large-scale model can be performed in combination with multiple GPUs.

[0022] By configuring multiple main processes and multiple sub-processes, distributed pre-training of large-scale models can be performed, which can improve the pre-training speed of large-scale models and shorten the pre-training time length of large-scale models.

[0023] In an embodiment of the present disclosure, the large-scale model pre-training process has multiple training cycles. In each training cycle, the main process is invoked to buffer the intermediate model parameters obtained by training in the main buffer. When buffering, the intermediate model parameters previously buffered in the main buffer are overwritten, thereby realizing the update process of the intermediate model parameters in the main buffer.

[0024] The intermediate model parameters in step 101 may be model parameters obtained by training in any one of the training cycles. For example, they may be model parameters obtained by training in the first training cycle or model parameters obtained by training in the last training cycle.

[0025] In the embodiments of the present disclosure, if the large-scale model is a mixed-precision large-scale model, that is, if the model parameters in the large-scale model include parameters of at least two precisions, such as 16-bit floating-point numbers and 32-bit floating-point numbers, the number of main buffers in one main process may be at least 2. The number of sub-buffers in a sub-process corresponding to the main process may be at least 2.

[0026] In one example, in one main process, the number of main buffers may match the number of floating-point precisions in the large-scale model, i.e., most significant floating-point numbers are stored in one main buffer, and each floating-point number in other digits is converted to a most significant floating-point number and stored in one main buffer.

[0027] In another example, the number of main buffers in one main process may be two, i.e., floating-point numbers with the most significant digit are stored in one main buffer, and floating-point numbers with other digits are converted to floating-point numbers with the most significant digit and stored in one main buffer.

[0028] In step 102, the sub-process is called via the main process to read the intermediate model parameters in the main buffer and perform parameter fusion processing based on the intermediate model parameters.

[0029] In an embodiment of the present disclosure, in order to realize direct reading processing of the intermediate model parameters in the main buffer by the sub-process and improve the efficiency of reading the intermediate model parameters, the sub-process can access the main buffer via an inter-process communication (IPC) mechanism to obtain the intermediate model parameters in the main buffer and store the intermediate model parameters in the sub-buffer.

[0030] An IPC mechanism, for example, a Compute Unified Device Architecture (CUDA) IPC mechanism, allows the sub-process to access the main buffer of the main process and read the intermediate model parameters in the main buffer.

[0031] In an embodiment of the present disclosure, the sub-buffer may be a high-speed memory in a central processing unit (CPU), and the main buffer may be a memory in a graphics processing unit (GPU). Faster data exchange between the memory in the GPU and the high-speed memory in the CPU can further improve the efficiency of the sub-process reading the intermediate model parameters.

[0032] Since the main process and sub-process are performed asynchronously, the sub-buffer is high-speed memory, and the main buffer is memory within the GPU, the sub-process can quickly read the main buffer of the main process. Furthermore, the sub-process can read the main buffer of the main process multiple times, so that the fusion process can be performed using intermediate model parameters from more training cycles, which further improves the model fusion efficiency and the accuracy of the fused model.

[0033] In the model fusion method according to the embodiment of the present disclosure, in the large-scale model pre-training process, a main process is called to buffer intermediate model parameters from the pre-training process in a main buffer, and a sub-process is called via the main process to read the intermediate model parameters from the main buffer and perform parameter fusion processing based on the intermediate model parameters. Here, since the main process and the sub-process can be performed asynchronously, parameter fusion processing can be performed on the intermediate model parameters obtained by pre-training the large-scale model while pre-training the large-scale model, thereby improving the efficiency of model fusion without affecting the efficiency of pre-training the large-scale model.

[0034] In order to improve the accuracy of the intermediate model parameters read by the sub-process, avoid simultaneously reading the intermediate model parameters obtained by training in two training cycles, and reduce the error rate of the read intermediate model parameters, the sub-process can be invoked to perform the reading process of the intermediate model parameters after one training cycle of the large-scale model is completed and the buffering of the intermediate model parameters obtained by training in the training cycle is completed. As shown in Figure 2, Figure 2 is a schematic diagram of a second embodiment of the present disclosure, and the embodiment shown in Figure 2 can include the following steps 201 and 202.

[0035] In step 201, in the large-scale model pre-training process, the main process is invoked to buffer the intermediate model parameters of the pre-training process into the main buffer. In step 202, every time pre-training of at least one training cycle of the large-scale model is completed, and when the buffering process of the intermediate model parameters obtained by training in the last training cycle of the at least one training cycle is completed, the intermediate model parameters in the main buffer are read by calling a sub-process via the main process, and parameter fusion processing is performed based on the intermediate model parameters.

[0036] In an embodiment of the present disclosure, in one example, each time pre-training of one training cycle of a large-scale model is completed and the buffering process of the intermediate model parameters obtained by training in that training cycle is completed, the intermediate model parameters in the main buffer can be read by calling a sub-process via the main process, and parameter fusion processing can be performed based on the intermediate model parameters.

[0037] In another example, each time pre-training of multiple training cycles of a large-scale model is completed, and when the buffering process of the intermediate model parameters obtained by training in the last training cycle of the multiple training cycles is completed, the intermediate model parameters in the main buffer can be read by calling a sub-process via the main process, and parameter fusion processing can be performed based on the intermediate model parameters.

[0038] In an embodiment of the present disclosure, in order to prevent the main process from re-buffering the intermediate model parameters in the main buffer if the sub-process has not finished reading the intermediate model parameters, the electronic device can determine the time length of a training cycle and set the time length to be equal to or greater than the time length for which the sub-process reads the intermediate model parameters in the main buffer, or set the time length for which the sub-process reads the intermediate model parameters in the main buffer to be equal to or less than the time length.

[0039] In an embodiment of the present disclosure, the number of training cycles in the at least one training cycle is determined based on the sum of the length of time the sub-process reads the intermediate model parameters and the length of time the sub-process performs parameter fusion processing based on the intermediate model parameters.

[0040] The time length for which the sub-process reads the intermediate model parameters refers to the time length required from the start of reading the intermediate model parameters by the sub-process to the completion of reading the intermediate model parameters. The time length for performing parameter fusion processing based on the intermediate model parameters refers to the time length required from the start of parameter fusion processing of the intermediate model parameters by the sub-process to the completion of the parameter fusion processing.

[0041] The electronic device can determine the sum of the length of time of a training cycle and the length of time for buffering the intermediate model parameters in the main buffer, set the sum as a first sum, set the sum of the length of time for the sub-process to read the intermediate model parameters and the length of time for performing parameter fusion processing based on the intermediate model parameters as a second sum, determine a quotient and remainder of the second sum and the first sum, and determine the sum of the quotient and 1 as the minimum number of training cycles in at least one training cycle.

[0042] The number of training cycles in at least one training cycle is determined based on the sum of the length of time the sub-process reads the intermediate model parameters and the length of time it performs parameter fusion processing based on the intermediate model parameters, thereby determining the minimum number of training cycles for which the sub-process can read one intermediate model parameter.The intermediate model parameters are read and fusion processing is performed based on this number.This allows the fusion processing to be performed using intermediate model parameters from more training cycles, achieving high-frequency model fusion processing, further improving the efficiency of model fusion and the accuracy of the fused model.

[0043] In an embodiment of the present disclosure, in order to further improve the accuracy of the fused model parameters, the manner in which the sub-process performs parameter fusion processing based on the intermediate model parameters includes: reading the historical fusion model parameters in the fusion sub-buffer, and the historical fusion model parameters are determined by fusing the historical intermediate model parameters obtained by training in at least two training cycles in the large-scale model pre-training process; performing a fusion process on the intermediate model parameters and the historical fusion model parameters to obtain current fusion model parameters; and storing the current fusion model parameters in the fusion sub-buffer.

[0044] The process in which the sub-process performs a fusion process on the intermediate model parameters and the historical fusion model parameters to obtain the current fusion model parameters may be, for example, determining a first weight for the intermediate model parameters and a second weight for the historical fusion model parameters, and performing a weighting and summation process on the intermediate model parameters and the historical fusion model parameters based on the first weight and the second weight to obtain the current fusion model parameters.

[0045] Since the fused model parameters are considered to be obtained by fusing intermediate model parameters from multiple training cycles, different weights can be set for the intermediate model parameters and the historical fused model parameters, which can further improve the accuracy of the fused current fused model parameters.

[0046] The sub-process can determine, for each parameter in the intermediate model parameters, a fusion parameter corresponding to the parameter in the history fusion model parameters, combine the first weight and the second weight, and perform weighting and summation on the parameter and the fusion parameter to obtain one fusion-processed parameter.

[0047] The fusion process of each parameter in the intermediate model parameters is independent and does not affect each other. The sub-process can execute the fusion process of each parameter in the intermediate model parameters in parallel, which further improves the efficiency of model fusion.

[0048] For details of step 201, reference can be made to step 101 in the embodiment shown in FIG. 1, and further detailed explanation will be omitted here.

[0049] In a model fusion method according to an embodiment of the present disclosure, during the pre-training process of a large-scale model, a main process is invoked to buffer intermediate model parameters from the pre-training process in a main buffer; each time at least one training cycle of the large-scale model is completed, and when the buffering process of the intermediate model parameters obtained by training in the last training cycle of the at least one training cycle is completed, a sub-process is invoked via the main process to read the intermediate model parameters from the main buffer and perform parameter fusion processing based on the intermediate model parameters. Here, after one training cycle of the large-scale model is completed and the buffering of the intermediate model parameters obtained by training in the training cycle is completed, the sub-process is invoked to read the intermediate model parameters, thereby avoiding the simultaneous reading of intermediate model parameters obtained by training in two training cycles and further improving the accuracy of the intermediate model parameters read by the sub-process.

[0050] When the number of sub-processes changes from a first number to a second number, the intermediate model parameters stored in the sub-buffer of each sub-process change, e.g., the number changes, and the sequence numbers of the stored intermediate model parameters also change. To avoid re-pre-training the large-scale model in this case and to avoid any impact on the efficiency of the pre-training process for the large-scale model in this case, a fusion process can be performed on the fusion model parameters of each sub-process when the number is the first and the fusion model parameters of each sub-process when the number is the second. As shown in FIG. 3, FIG. 3 is a schematic diagram according to a third embodiment of the present disclosure. The embodiment shown in FIG. 3 can include the following steps 301 to 305.

[0051] In step 301, in the large-scale model pre-training process, the main process is invoked to buffer the intermediate model parameters of the pre-training process into the main buffer. In step 302, the sub-process is called via the main process to read the intermediate model parameters in the main buffer and perform parameter fusion processing based on the intermediate model parameters. In step 303, when the number of sub-processes changes from the first number to the second number, a first fusion model parameter of each sub-process when the number of sub-processes is the first number is obtained, and the maximum value of the sequence number of each training cycle corresponding to the fused intermediate model parameter in the first fusion model parameter is N.

[0052] In the embodiment of the present disclosure, sub-processes correspond one-to-one to main processes. When the number of sub-processes changes from a first number to a second number, the number of main processes also changes from the first number to the second number. When main buffers corresponding to multiple main processes are located on different GPUs, the number of GPUs in the large-scale model pre-training process also changes. That is, card expansion and contraction exist in the large-scale model pre-training process. Card expansion refers to an increase in the number of GPUs, and card contraction refers to a decrease in the number of GPUs. When cards are expanded and contracted, the number of sub-processes also changes.

[0053] The maximum value of the sequence number of each training cycle corresponding to the fused intermediate model parameters in the first fusion model parameters is N, which means that the previous parameter fusion process is a fusion process for the intermediate model parameters and history fusion model parameters obtained by training in the Nth training cycle.

[0054] In step 304, obtain second fusion model parameters for each sub-process when the number of sub-processes is a second number, and the second fusion model parameters are obtained by fusing intermediate model parameters from the N+1th training cycle to the tth training cycle. N is a positive integer of 1 or greater, and t is a positive integer of 1 or greater.

[0055] In step 305, a fusion process is performed on each of the first fusion model parameters and each of the second fusion model parameters to obtain fused parameters.

[0056] In an embodiment of the present disclosure, since the total parameter amount of each first fusion model parameter and each second fusion model parameter is relatively large, in order to easily perform fusion processing on each first fusion model parameter and each second fusion model parameter, each first fusion model parameter, each second fusion parameter and the fused parameters can be stored on a hard disk during the fusion processing.

[0057] In an embodiment of the present disclosure, the process in which the electronic device performs step 305 may be, for example, performing a combination process on each first fusion model parameter to obtain a first combination parameter of the large-scale model, performing a combination process on each second fusion model parameter to obtain a second combination parameter of the large-scale model, determining a third weight for the first combination parameter based on N and t, and performing a fusion process on the first combination parameter and the second combination parameter based on the third weight to obtain a fused parameter.

[0058] "Performing a combination process on each first fusion model parameter" can refer to performing a configuration process on each parameter in each first fusion model parameter according to the parameter configuration method in the large-scale model. "Performing a combination process on each second fusion model parameter" can refer to performing a configuration process on each parameter in each second fusion model parameter according to the parameter configuration method in the large-scale model. The first and second combination parameters may be stored on a hard disk, for example, in H5 format.

[0059] The value of the third weight is mainly affected by N and t, and determining the third weight based on N and t can improve the accuracy of determining the third weight. Furthermore, combining the third weight and performing a fusion process on the first combined parameter and the second combined parameter can further improve the accuracy of the fused parameter and eliminate the impact of changes in the number of sub-processes on model fusion.

[0060] In an embodiment of the present disclosure, the process by which the electronic device determines the third weight may be, for example, determining a difference value between t and N, and determining the third weight based on the difference value and the second weight of the history fusion model parameter in the sub-buffer of the sub-process.

[0061] The third weight may be a value whose base is the second weight and whose exponent is the difference value. The fact that the third weight is a value with the second weight as the base and the difference value as the exponent can be verified by combining a model fusion validation process. The validation process may be performed, for example, by generating intermediate model parameters shown in Equation (1) through multiple training cycles in a large-scale model pre-training process, and then verifying M nA fault may occur in , the training may be interrupted, and the fusion sub-buffer may be restarted. Here, the formula for the parameter fusion process may be as shown in Equation (3). Equation (2) indicates that the initial model parameters of the large-scale model are all 0. M={M0,M1,…,M n ,M n+1 ,…,M t} (1) M0=0 (2) M t ′ =αM t-1 ′ +(1-α)M t (3) where M represents the sequence of intermediate model parameters over multiple training cycles, and M0 represents the initial model parameters of the large-scale model. n represents the intermediate model parameters in the Nth training cycle. M t represents the intermediate model parameters in the t-th training cycle. α is the weight of the history fusion model parameters. (1-α) is the weight of the intermediate model parameters. M t ′ denotes the history fusion model parameters after the t-th training cycle.

[0062] Expanding equation (3) yields equation (4) below.

number

[0063] Since the training is stopped for the Nth training cycle, we actually obtained two sets of fused model parameters. n ′ represents the first set of fusion model parameters, and M t ″ represents the second set of fusion model parameters.

[0064] The calculation formulas for the two sets of fusion model parameters can be shown as the following formulas (5) and (6), respectively.

number

[0065] The equation to the right of the equal sign in equation (4) can be separated into two parts.

number

[0066] By using the second weight as the base and the difference value as the exponent, a third weight can be obtained, and further, by combining the third weight and performing a fusion process on the first combined parameter and the second combined parameter, the accuracy of the fused parameter can be further improved.

[0067] In an embodiment of the present disclosure, after step 305, in order to ensure the accuracy of the fused model parameters in the fusion sub-buffer of each sub-process, the electronic device may further perform a process of distributing and storing each parameter in the fused parameters in the fusion sub-buffer of each sub-process based on the second number of sub-processes.

[0068] In an embodiment of the present disclosure, the process in which the electronic device distributes and stores each parameter in the fused parameters in the fusion sub-buffers of each sub-process based on the second number of sub-processes may, for example, be to determine, for each sub-process, a sequence number of each parameter buffered in the fusion sub-buffer of the sub-process based on the second number of sub-processes, select a target fusion parameter from the fused parameters based on the sequence number, and store the target fusion parameter in the fusion sub-buffer of the sub-process.

[0069] Fusion parameters with different sequence numbers are buffered in the fusion sub-buffers of different sub-processes. Intermediate model parameters with different sequence numbers are buffered in the sub-buffers of different sub-processes. Based on the sequence numbers, it is possible to distinguish which parameters in the fused parameters need to be stored in which fusion sub-buffers of the sub-processes, thereby improving the accuracy of distributing and storing each parameter in the fused parameters.

[0070] For details of steps 301 and 302, reference can be made to steps 101 and 102 in the embodiment shown in FIG. 1, and further detailed explanations will be omitted here.

[0071] In a model fusion method according to an embodiment of the present disclosure, in a pre-training process for a large-scale model, a main process is called to buffer intermediate model parameters of the pre-training process in a main buffer, and sub-processes are called via the main process to read the intermediate model parameters in the main buffer. A parameter fusion process is performed based on the intermediate model parameters. When the number of sub-processes changes from a first number to a second number, first fusion model parameters are obtained for each sub-process when the number of sub-processes is the first number. The maximum value of the sequence numbers of each training cycle corresponding to the fused intermediate model parameters in the first fusion model parameters is N, and second fusion model parameters are obtained for each sub-process when the number of sub-processes is the second number. The second fusion model parameters are obtained by fusing the intermediate model parameters from the (N+1)th training cycle to the tth training cycle. A fusion process is performed on each of the first fusion model parameters and each of the second fusion model parameters to obtain fused parameters. When the number of processes changes, a fusion process is performed on each of the first fusion model parameters before the change and each of the second fusion model parameters after the change. This makes it possible to avoid interruptions in the pre-training of large-scale models and the model fusion process, thereby further improving the efficiency of the pre-training process of large-scale models and the efficiency of the model fusion process.

[0072] An example will be given below. Figure 4 is a schematic diagram of the framework of main processes and sub-processes. Figure 4 takes one main process and its corresponding sub-processes as an example. Main Process represents the main process, and ZCC Process represents the sub-process.

[0073] Model Param1 and Model Param2 of the Main Process represent intermediate model parameters with floating-point numbers in the most significant digits and are stored in the main buffer (Model Comm Buffer) of the Main Process. Optimizer Param1, Optimizer Param2, Optimizer State1, and Optimizer State2 represent intermediate model parameters with floating-point numbers in relatively less significant digits and are stored in the main buffer (Fused Optimizer Buffer) of the Main Process.

[0074] In Figure 4, the dark gray Model Comm Buffer of the Main Process and the dark gray Model Comm Buffer of the ZCC Process both represent the same main buffer. The dark gray Fused Optimizer Buffer of the Main Process and the dark gray Fused Optimizer Buffer of the ZCC Process both represent the same main buffer.

[0075] In Figure 4, the light grey Model Comm Buffer of the ZCC Process represents one sub-buffer of the ZCC Process. The light grey Fused Optimizer Buffer of the ZCC Process represents another sub-buffer of the ZCC Process.

[0076] The ZCC Process can read intermediate model parameters from the dark gray Model Comm Buffer of the Main Process via the IPC mechanism and store the read intermediate model parameters in the light gray Model Comm Buffer of the ZCC Process.The ZCC Process can read intermediate model parameters from the dark gray Fused Optimizer Buffer of the Main Process via the IPC mechanism and store the read intermediate model parameters in the light gray Fused Optimizer Buffer of the ZCC Process.

[0077] In Figure 4, the light gray Model EMA Buffer of the ZCC Process represents one fusion sub-buffer of the ZCC Process, which corresponds to the light gray Model Comm Buffer of the ZCC Process and is used to perform parameter fusion processing based on the intermediate model parameters in the light gray Model Comm Buffer of the ZCC Process and store the fused model parameters.

[0078] In Fig. 4, the light gray Fused EMA Optimizer Buffer of the ZCC Process represents another fusion sub-buffer of the ZCC Process, which corresponds to the light gray Fused Optimizer Buffer of the ZCC Process and is used to perform parameter fusion processing based on the intermediate model parameters in the light gray Fused Optimizer Buffer of the ZCC Process and store the fused model parameters.

[0079] The dark gray Model Comm Buffer and dark gray Fused Optimizer Buffer of the Main Process may be CUDA memory in the graphics processing unit (GPU), and the light gray Model Comm Buffer and light gray Fused Optimizer Buffer of the ZCC Process may be high-speed memory in the central processing unit (CPU).

[0080] An example will be given below. Figure 5 shows a schematic diagram of the execution of the main process and sub-processes. Figure 5 takes one main process and its corresponding sub-processes as an example. Forward / Backward represents one training cycle in the large-scale model pre-training process, i.e., forward propagation / backward propagation within one training cycle. Step represents the number of training cycles in which intermediate model parameters are stored in the main buffer. Here, Fwd / Bwd are abbreviations of Forward / Backward.

[0081] In Figure 5, "offload" indicates that a subprocess reads intermediate model parameters from the main buffer. "Update IPC" indicates that a subprocess accesses the main buffer. "EMA&Dump" indicates that the subprocess reads intermediate model parameters and performs parameter fusion processing.

[0082] In FIG. 5, the duration of one training cycle is equal to or greater than the duration of the sub-process reading the intermediate model parameters in the main buffer, i.e., equal to or greater than the sum of the duration of Update IPC and the duration of Offload.

[0083] To realize the above embodiment, the present disclosure further provides a model fusion device. As shown in Fig. 6, Fig. 6 is a schematic diagram of a fourth embodiment of the present disclosure. The model fusion device 60 can include a first calling module 601 and a second calling module 602.

[0084] The first calling module 601 is configured to buffer intermediate model parameters of the large-scale model pre-training process in a main buffer by calling a main process, and the second calling module 602 is configured to read the intermediate model parameters in the main buffer by calling a sub-process via the main process, and perform parameter fusion processing based on the intermediate model parameters.

[0085] In one possible implementation form of the embodiment of the present disclosure, the number of the main processes is multiple, the intermediate model parameters stored in the main buffer corresponding to each of the main processes are local model parameters within the large-scale model, and the number of the sub-processes is multiple, and the main processes and the sub-processes have a one-to-one correspondence.

[0086] In one possible implementation of an embodiment of the present disclosure, the condition for calling the sub-process is that each time pre-training of at least one training cycle of the large-scale model is completed, and the buffering process of intermediate model parameters obtained by training in the last training cycle of the at least one training cycle is completed.

[0087] In one possible implementation form of an embodiment of the present disclosure, the number of training cycles in the at least one training cycle is determined based on the sum of the length of time the sub-process reads the intermediate model parameters and the length of time it performs parameter fusion processing based on the intermediate model parameters.

[0088] In one possible implementation form of an embodiment of the present disclosure, the sum of the length of time of the training cycle and the length of time for buffering the intermediate model parameters in the main buffer is equal to or greater than the length of time for the sub-process to read the intermediate model parameters in the main buffer.

[0089] As one possible implementation form of an embodiment of the present disclosure, the manner in which the sub-process reads the intermediate model parameters in the main buffer includes accessing the main buffer via an inter-process communication mechanism to obtain the intermediate model parameters in the main buffer, and storing the intermediate model parameters in a sub-buffer.

[0090] In one possible implementation of the embodiment of the present disclosure, the sub-buffer is a high-speed memory in a central processing unit (CPU), and the main buffer is a memory in the graphics processing unit (GPU).

[0091] As one possible implementation form of the embodiment of the present disclosure, the manner in which the sub-process performs parameter fusion processing based on the intermediate model parameters includes: reading historical fusion model parameters in a fusion sub-buffer, where the historical fusion model parameters are determined by fusing historical intermediate model parameters obtained by training in at least two training cycles in the large-scale model pre-training process; performing a fusion processing on the intermediate model parameters and the historical fusion model parameters to obtain current fusion model parameters; and storing the current fusion model parameters in the fusion sub-buffer.

[0092] In one possible implementation form of the embodiment of the present disclosure, the manner in which the sub-process performs parameter fusion processing based on the intermediate model parameters further includes: determining a first weight for the intermediate model parameters and a second weight for the history fusion model parameters; and performing a weighting and summation processing on the intermediate model parameters and the history fusion model parameters based on the first weight and the second weight to obtain the current fusion model parameters.

[0093] In one possible implementation of the embodiment of the present disclosure, the model fusion device further includes a first acquisition module, a second acquisition module, and a fusion processing module. The first acquisition module is configured to acquire first fusion model parameters for each sub-process when the number of sub-processes changes from a first number to a second number, where the maximum value of sequence numbers of training cycles corresponding to fused intermediate model parameters in the first fusion model parameters is N. The second acquisition module is configured to acquire second fusion model parameters for each sub-process when the number of sub-processes changes from a first number to a second number, where the second fusion model parameters are acquired by fusing intermediate model parameters from the (N+1)th training cycle to the tth training cycle. The fusion processing module is configured to perform a fusion process on each of the first fusion model parameters and each of the second fusion model parameters to acquire fused parameters.

[0094] As one possible implementation form of an embodiment of the present disclosure, the fusion processing module is configured to perform a combination process on each of the first fusion model parameters to obtain a first combination parameter of the large-scale model, perform a combination process on each of the second fusion model parameters to obtain a second combination parameter of the large-scale model, determine a third weight for the first combination parameter based on N and t, and perform a fusion process on the first combination parameter and the second combination parameter based on the third weight to obtain the fused parameter.

[0095] As one possible implementation form of the embodiment of the present disclosure, the fusion processing module is further configured to determine a difference value between the t and the N, and determine the third weight based on the difference value and a second weight of the history fusion model parameter in the sub-buffer of the sub-process.

[0096] In one possible implementation form of the embodiment of the present disclosure, the third weight is a value whose base is the second weight and whose exponent is the difference value.

[0097] As one possible implementation form of an embodiment of the present disclosure, the model fusion device further includes a storage module configured to distribute and store each parameter in the fused parameters in fusion sub-buffers of each of the sub-processes based on a second number of the sub-processes.

[0098] As one possible implementation form of an embodiment of the present disclosure, the storage module is configured to, for each sub-process, determine a sequence number of each parameter buffered in the fusion sub-buffer of the sub-process based on the second number of the sub-processes, select a target fusion parameter from the fused parameters based on the sequence number, and store the target fusion parameter in the fusion sub-buffer of the sub-process.

[0099] In one possible implementation of the embodiments of the present disclosure, during the fusion process, each of the first fusion model parameters, each of the second fusion parameters and the fused parameters are stored on a hard disk.

[0100] In the model fusion device according to the embodiment of the present disclosure, in the pre-training process of a large-scale model, a main process is called to buffer intermediate model parameters from the pre-training process in a main buffer, and a sub-process is called via the main process to read the intermediate model parameters from the main buffer and perform parameter fusion processing based on the intermediate model parameters. Here, since the main process and the sub-process can be performed asynchronously, parameter fusion processing can be performed on the intermediate model parameters obtained by pre-training the large-scale model while pre-training the large-scale model, thereby improving the efficiency of model fusion without affecting the efficiency of pre-training the large-scale model.

[0101] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision and disclosure of users' personal information are all carried out with the users' consent, are in accordance with the provisions of relevant laws and regulations, and are not contrary to public order and morals.

[0102] According to an embodiment of the present disclosure, the present disclosure further provides an electronic device, a readable storage medium, and a computer program.

[0103] 7 illustrates a schematic block diagram of an exemplary electronic device 700 according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital assistants, mobile phones, smartphones, wearable devices, and other similar computing devices. The components, their connections and relationships, and their functions illustrated herein are merely examples and are not intended to limit the practice of the present disclosure as described and / or claimed herein.

[0104] 7, device 700 includes a computing unit 701 that can perform various appropriate operations and processes based on a computer program stored in a read-only memory (ROM) 702 or loaded from a storage unit 708 into a random access memory (RAM) 703. The RAM 703 can also store various programs and data necessary for the operation of device 700. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.

[0105] The components in device 700 are connected to an I / O interface 705, which includes an input unit 706 such as a keyboard, a mouse, etc., an output unit 707 such as various displays, speakers, etc., a storage unit 708 such as a magnetic disk, an optical disk, etc., and a communication unit 709 such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows device 700 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0106] The computing unit 701 may be various general-purpose and / or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that execute machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 701 performs various methods and processes described above, such as the model fusion method. For example, in some embodiments, the model fusion method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 708. In some embodiments, some or all of the computer program may be loaded and / or installed into the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, it may perform one or more steps of the model fusion method described above. Alternatively, in other embodiments, the computing unit 701 may be configured to execute the model fusion method in any other suitable manner (e.g., by firmware).

[0107] Various embodiments of the systems and techniques described herein may be realized in digital electronic circuitry systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may be embodied in one or more computer programs that can be executed and / or interpreted by a programmable system including at least one programmable processor. The programmable processor, which may be a special-purpose or general-purpose programmable processor, can receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0108] Program codes for implementing the methods of the present disclosure can be written using any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, and when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are performed. The program code can be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0109] In the context of this disclosure, a machine-readable medium may be a tangible medium that can contain or store a program used by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. The machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the above. More specific examples of machine-readable storage media include one or more line-based electrical connections, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0110] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user, and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other types of devices can provide interaction with a user; for example, the feedback provided to the user can be any form of sensing feedback (e.g., visual feedback, auditory feedback, or tactile feedback) and can receive input from the user in any form (including acoustic, speech, or tactile input).

[0111] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user interacts with embodiments of the systems and techniques described herein), or any combination of such back-end, middleware, and front-end components. The components of the system may be interconnected via any form or medium of digital data communication (e.g., a communications network). Examples of communications networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0112] The computer system may include a client and a server. The client and the server are generally remote from each other and typically interact via a communication network. The relationship between the client and the server is created by computer programs running on corresponding computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server in combination with a blockchain.

[0113] The various forms of flow shown above may be used to rearrange, add, or delete steps. For example, the steps described in this disclosure may be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solution disclosed in this disclosure are achieved, and this specification is not limited thereto.

[0114] The above specific implementation modes do not limit the protection scope of the present disclosure. Those skilled in the art can make various modifications, combinations, subcombinations, and substitutions based on design requirements and other factors. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present disclosure should be included within the protection scope of the present disclosure.

Claims

1. 1. A model fusion method, comprising: In a large-scale model pre-training process, buffering intermediate model parameters of the pre-training process into a main buffer by calling a main process; reading the intermediate model parameters in the main buffer by calling a sub-process via the main process, and performing a parameter fusion process based on the intermediate model parameters; A model fusion method comprising:

2. the number of the main processes is plural, and the intermediate model parameters stored in the main buffer corresponding to each of the main processes are local model parameters within the large-scale model; The model fusion method according to claim 1 , wherein the number of the sub-processes is plural, and the main process and the sub-process correspond to each other one-to-one.

3. 2. The model fusion method of claim 1, wherein the subprocess is called upon each time pre-training of at least one training cycle of the large-scale model is completed and upon completion of buffering of intermediate model parameters obtained by training in the last training cycle of the at least one training cycle.

4. 4. The model fusion method of claim 3, wherein the number of training cycles in the at least one training cycle is determined based on a sum of a length of time for the sub-process to read the intermediate model parameters and a length of time for the sub-process to perform parameter fusion processing based on the intermediate model parameters.

5. 4. The method of claim 3, wherein the length of time of the training cycle is equal to or greater than the length of time that the sub-process reads the intermediate model parameters in the main buffer.

6. The sub-process reads the intermediate model parameters in the main buffer in the following manner: accessing the main buffer via an inter-process communication mechanism to obtain the intermediate model parameters in the main buffer; storing the intermediate model parameters in a sub-buffer; The model fusion method of claim 1 , comprising:

7. The sub-buffer is a high-speed memory within a central processing unit (CPU), The model fusion method of claim 6 , wherein the main buffer is a memory within the graphics processing unit (GPU).

8. The sub-process performs parameter fusion processing based on the intermediate model parameters, Reading historical fusion model parameters in a fusion sub-buffer, the historical fusion model parameters being determined based on a fusion of historical intermediate model parameters obtained by training in at least two training cycles in a pre-training process of the large-scale model; performing a fusion process on the intermediate model parameters and the history fusion model parameters to obtain current fusion model parameters; storing the current fused model parameters in the fused sub-buffer; The model fusion method of claim 1 , comprising:

9. performing a fusion process on the intermediate model parameters and the history fusion model parameters to obtain current fusion model parameters, determining first weights for the intermediate model parameters and second weights for the history fusion model parameters; performing a weighting and summing process on the intermediate model parameters and the historical fusion model parameters based on the first weight and the second weight to obtain the current fusion model parameters; The model fusion method of claim 8 , comprising:

10. The model fusion method includes: When the number of the sub-processes changes from a first number to a second number, a step of obtaining a first fusion model parameter of each sub-process when the number of the sub-processes is the first number, wherein the maximum value of the sequence number of each training cycle corresponding to the fused intermediate model parameter in the first fusion model parameter is N; obtaining second fusion model parameters for each sub-process when the number of the sub-processes is a second number, wherein the second fusion model parameters are obtained by fusing intermediate model parameters from the (N+1)th training cycle to the tth training cycle; performing a fusion process on each of the first fusion model parameters and each of the second fusion model parameters to obtain fused parameters; The model fusion method of claim 2 further comprising:

11. The step of performing a fusion process on each of the first fusion model parameters and each of the second fusion model parameters includes: performing a combination process on each of the first fusion model parameters to obtain a first combination parameter of the large-scale model; performing a combination process on each of the second fusion model parameters to obtain second combination parameters of the large-scale model; determining a third weight for the first combination parameter based on N and t; performing a fusion process on the first combined parameter and the second combined parameter based on the third weight to obtain the fused parameter; The model fusion method of claim 10, comprising:

12. determining a third weight of the first combination parameter based on N and t, determining a difference between t and N; determining the third weight based on the difference value and a second weight of the history fusion model parameter in the sub-buffer of the sub-process; The model fusion method of claim 11 , comprising:

13. The model fusion method according to claim 12 , wherein the third weight is a value whose base is the second weight and whose exponent is the difference value.

14. The model fusion method includes: The model fusion method according to claim 10 , further comprising the step of distributing and storing each parameter in the fused parameters in fusion sub-buffers of each of the sub-processes based on the second number of the sub-processes.

15. The step of distributing and storing each parameter in the fused parameters in fused sub-buffers of each of the sub-processes based on the second number of the sub-processes includes: determining, for each sub-process, a sequence number for each parameter buffered in the fusion sub-buffer of the sub-process based on a second number of the sub-processes; selecting a target fusion parameter from the fused parameters based on the sequence number; storing the target fusion parameters in a fusion sub-buffer of the sub-process; The model fusion method of claim 14, comprising:

16. The model fusion method of claim 10 , wherein during the fusion process, each of the first fusion model parameters, each of the second fusion parameters and the fused parameters are stored on a hard disk.

17. A model fusion device, A first calling module is configured to, in a large-scale model pre-training process, call a main process to buffer intermediate model parameters of the pre-training process in a main buffer; a second calling module configured to read the intermediate model parameters in the main buffer by calling a sub-process via the main process, and perform a parameter fusion process based on the intermediate model parameters; A model fusion device comprising:

18. 1. An electronic device comprising: at least one processor; a memory communicatively coupled to the at least one processor; 17. An electronic device, wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to cause the at least one processor to perform the model fusion method of any one of claims 1 to 16.

19. A non-transitory computer-readable storage medium having computer instructions stored thereon, comprising: A non-transitory computer readable storage medium, the computer instructions being used to cause a computer to perform the model fusion method of any one of claims 1 to 16.

20. A computer program comprising: A computer program product, which when executed by a processor, implements the model fusion method of any one of claims 1 to 16.

Images