Model training method, model training device, electronic device, and storage medium

Abstract

Embodiments of the present disclosure provide a model training method, a model training device, an electronic device and a storage medium. The model training method comprises: in response to obtaining an insertion layer configuration corresponding to a first model, determining insertion layer information according to the insertion layer configuration, wherein the insertion layer configuration comprises parallel strategy information, and the insertion layer information comprises an insertion layer number, an insertion layer type and insertion layer position information; updating the first model according to the insertion layer information to obtain a second model; and performing staggered pipeline parallel training on the second model. The method can decouple from runtime performance measurement, avoid complex search and analysis, ensure the availability of staggered pipeline scheduling in the composition stage, improve system flexibility and compatibility, support high-performance training under different model structures and different parallel configurations, and improve training efficiency.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of artificial intelligence, specifically to a model training method, a model training apparatus, an electronic device, and a storage medium. Background Technology

[0002] In the field of deep learning, model training typically relies on high-performance computing devices to accelerate large-scale computations. However, due to hardware limitations, the memory capacity of a single computing device is limited, making it difficult to support models with a large number of parameters (such as models based on the Transformer architecture). To overcome the memory bottleneck and achieve efficient training, pipeline parallelism (PP) has become the mainstream distributed training strategy. Within the pipeline parallel framework, how to further improve training efficiency has become a pressing issue. Summary of the Invention

[0003] At least one embodiment of this disclosure provides a model training method, comprising: in response to obtaining an insertion layer configuration corresponding to a first model, determining insertion layer information according to the insertion layer configuration, wherein the insertion layer configuration includes parallel strategy information, and the insertion layer information includes the number of insertion layers, the type of insertion layer, and the position information of the insertion layer; updating the first model according to the insertion layer information to obtain a second model; and performing interleaved pipelined parallel training on the second model.

[0004] In the model training method provided in at least one embodiment of this disclosure, the parallel strategy information includes pipeline parallelism, the number of virtual stages, and the total number of layers of the first model. The step of determining the insertion layer information based on the insertion layer configuration in response to obtaining the insertion layer configuration corresponding to the first model includes: determining the number of insertion layers based on the pipeline parallelism, the number of virtual stages, and the total number of layers of the first model; and determining the insertion layer position information based on the total number of layers of the first model and the number of insertion layers in response to the insertion layer number not being equal to 0.

[0005] In at least one embodiment of the model training method provided in this disclosure, determining the number of insertion layers based on the pipeline parallelism, the number of virtual stages, and the total number of layers of the first model includes: determining the number of insertion layers based on the product of the pipeline parallelism and the number of virtual stages, and the total number of layers of the first model.

[0006] In at least one embodiment of the model training method provided in this disclosure, determining the insertion layer position information based on the total number of layers of the first model and the number of insertion layers includes: determining an insertion interval value based on the total number of layers of the first model and the number of insertion layers; determining a first number of insertion position indices based on the insertion interval value to obtain the insertion layer position information, wherein the first number is equal to the number of insertion layers.

[0007] In the model training method provided in at least one embodiment of this disclosure, the insertion layer configuration further includes at least one of the following: insertion layer enable configuration, insertion layer type configuration, insertion layer position configuration, or custom insertion layer configuration, wherein the insertion layer enable configuration is used to determine whether to enable the insertion strategy, wherein the step of determining the insertion layer information according to the insertion layer configuration in response to obtaining the insertion layer configuration corresponding to the first model further includes: determining the insertion layer type according to the insertion layer type configuration in response to the insertion layer configuration including the insertion layer type configuration; determining the insertion layer position information according to the insertion layer position configuration in response to the insertion layer configuration including the insertion layer position configuration and the insertion layer position configuration conforming to a first preset rule, wherein after determining the insertion layer information according to the insertion layer configuration, the model training method further includes: updating the insertion layer information based on the custom insertion layer configuration in response to the insertion layer configuration including the custom insertion layer configuration and the custom insertion layer configuration conforming to a second preset rule.

[0008] In the model training method provided in at least one embodiment of this disclosure, the insertion layer type includes at least one of the following: identity placeholder layer, copy layer, control time consumption layer, or control memory layer.

[0009] In the model training method provided in at least one embodiment of this disclosure, the insertion layer position information includes an insertion position index and a relative position marker.

[0010] In at least one embodiment of the model training method provided in this disclosure, updating the first model according to the insertion layer information includes: generating a corresponding insertion layer according to the insertion layer information; inserting the insertion layer into a specified position in the first model, wherein the specified position is determined according to the insertion layer position information.

[0011] In the model training method provided in at least one embodiment of this disclosure, the step of performing interleaved pipelined parallel training on the second model includes: dividing the second model into multiple layers according to the number of virtual stages and the degree of pipeline parallelism, and mapping the multiple layers to corresponding computing devices for interleaved pipelined parallel training.

[0012] In at least one embodiment of the model training method provided in this disclosure, after the second model is trained in an interleaved pipeline parallel manner, the model training method further includes: performing an insertion layer deletion operation on the trained second model according to the insertion layer information to obtain the first model carrying weights; and saving the weights of the first model.

[0013] In at least one embodiment of the model training method provided in this disclosure, before performing interleaved pipelined parallel training on the second model, the model training method further includes: performing an insertion layer deletion operation on the second model according to the insertion layer information to obtain the first model; loading pre-stored weights that match the structure of the first model into the first model; and updating the first model carrying the weights according to the insertion layer information to obtain the second model carrying the weights.

[0014] At least one embodiment of this disclosure provides a model training apparatus, comprising: a determining module configured to determine insert layer information based on the insert layer configuration according to an obtained insert layer configuration corresponding to a first model, wherein the insert layer configuration includes parallel strategy information, and the insert layer information includes the number of insert layers, the type of insert layer, and the position information of the insert layer; an updating module configured to update the first model according to the insert layer information to obtain a second model; and a training module configured to perform interleaved pipeline parallel training on the second model.

[0015] At least one embodiment of this disclosure provides an electronic device, including: at least one processor; at least one memory including one or more computer program modules; wherein the one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the model training method provided in at least one embodiment of this disclosure.

[0016] At least one embodiment of this disclosure provides a non-transitory computer-readable storage medium storing computer-readable instructions thereon, wherein the computer-readable instructions, when executed by at least one processor, perform the model training method provided in at least one embodiment of this disclosure.

[0017] This disclosure provides at least one embodiment of a model training method, model training apparatus, electronic device, and non-transitory computer-readable storage medium, proposing an interpolation layer-driven model construction scheme. This scheme introduces an interpolation layer to rewrite the model structure during the graph construction phase (i.e., the model building phase), thereby decoupling it from runtime performance metrics, avoiding complex search and analysis, and ensuring the availability of interleaved pipeline scheduling during the graph construction phase. Furthermore, it can simultaneously support non-uniform and non-continuous stage mapping. This method overcomes the rigid limitations of traditional distributed frameworks on integer constraints, improves system flexibility and compatibility, supports high-performance training under different model structures and different parallel configurations, and improves training efficiency. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings of the embodiments will be briefly described below. Obviously, the drawings described below only relate to some embodiments of this disclosure and are not intended to limit this disclosure.

[0019] Figure 1A This is a schematic diagram of a non-interleaved pipeline parallel training process;

[0020] Figure 1B This is a schematic diagram of an interleaved pipeline parallel training process;

[0021] Figure 2 A flowchart illustrating a model training method provided in at least one embodiment of this disclosure;

[0022] Figure 3 A flowchart illustrating a model training method provided in at least one embodiment of this disclosure;

[0023] Figure 4 A flowchart illustrating a model training method provided in at least one embodiment of this disclosure;

[0024] Figure 5 An exemplary schematic diagram of a weight storage method provided in at least one embodiment of this disclosure;

[0025] Figure 6 A schematic block diagram of a model training apparatus provided in at least one embodiment of this disclosure;

[0026] Figure 7 A schematic block diagram of an electronic device provided for at least one embodiment of this disclosure;

[0027] Figure 8 A schematic block diagram of another electronic device provided for at least one embodiment of this disclosure;

[0028] Figure 9This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this disclosure. All other embodiments obtained by those skilled in the art based on the described embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0030] Unless otherwise defined, the technical or scientific terms used in this disclosure shall have the ordinary meaning understood by one of ordinary skill in the art to which this disclosure pertains. The terms “first,” “second,” and similar terms used in this disclosure do not indicate any order, quantity, or importance, but are merely used to distinguish different components. Terms such as “comprising” or “including” mean that the element or object preceding the word encompasses the elements or objects listed following the word and their equivalents, without excluding other elements or objects. Terms such as “connected” or “linked” are not limited to physical or mechanical connections, but can include electrical connections, whether direct or indirect. Terms such as “upper,” “lower,” “left,” and “right” are used only to indicate relative positional relationships, and these relative positional relationships may change accordingly when the absolute position of the described objects changes.

[0031] The present disclosure will now be described through several specific embodiments. To keep the following description of the embodiments of the present disclosure clear and concise, detailed descriptions of known functions and known components may be omitted. When any component of an embodiment of the present disclosure appears in more than one drawing, the component is represented by the same or similar reference numerals in each drawing.

[0032] Pipeline parallelism is a crucial technique for training large-scale deep learning models, such as those based on the Transformer architecture. In pipeline parallelism, the model's network layers are sequentially divided into multiple stages, each containing several consecutive layers, and these stages are distributed across different computing devices, forming a cross-device computational pipeline. For example, during forward propagation, each computing device passes activation values ​​to the next pipeline stage, and during backward propagation, each computing device transmits gradients back to the previous pipeline stage.

[0033] To improve pipeline throughput, a global training batch is typically divided into multiple micro-batches. These micro-batches flow sequentially into the pipeline, progressing step-by-step between different stages, forming a pipeline-like execution pattern. As the number of micro-batches increases, the pipeline approaches full capacity, and pipeline bubbles (i.e., idle time for computing devices waiting for data) are compressed, significantly improving computational efficiency. Through this mechanism, multiple computing devices can concurrently process data from different micro-batches within the same training cycle, achieving computational overlap in the time dimension and effectively improving hardware resource utilization and overall training efficiency.

[0034] Figure 1A This is a schematic diagram of a non-interleaved pipeline parallel training process.

[0035] Figure 1A The pipelined parallel training process illustrated employs a One Forward pass followed by One Backward pass (1F1B) scheduling strategy. The core idea of ​​the 1F1B strategy is that within each pipeline stage, after completing the forward propagation computation of a micro-batch sequentially, the corresponding backward propagation computation immediately begins, without waiting for all micro-batches to complete their forward propagation, thus improving training efficiency and throughput. Under this scheduling mechanism, the computing devices in each stage immediately begin processing and generating output after receiving the output from the previous stage, and then pass it to the computing devices in the next stage. However, in a non-interleaved pipeline architecture, each computing device is only responsible for one consecutive stage, making it difficult to dynamically adjust the computational load. When the computational load is uneven across stages, some computing devices may become overloaded while others remain idle, affecting the overall system performance. Furthermore, since the task execution between stages is sequential, significant idle times occur at the beginning and end of a complete pipeline cycle. This means that during certain time periods, some computing resources are idle and not fully utilized, thus reducing overall training efficiency.

[0036] For example, such as Figure 1AAs shown, the target model's multiple layers are divided into four stages (stages A through D, not shown in the figure), which are assigned to computing devices A through D for computation. Each stage includes, for example, eight micro-batches of data. Fi in the row corresponding to computing device A represents the forward propagation of the i-th micro-batch of data in stage A, Bx in the row corresponding to computing device D represents the backward propagation of the x-th micro-batch of data in stage D, and so on. During forward computation, the computation of the next stage depends on the computation of the previous stage. For example, the forward propagation F1 of the first micro-batch of data on computing device B (corresponding to stage B) can only begin after the F1 computation on computing device A (corresponding to stage A) is completed. Similarly, during backward computation, the computation of the previous stage depends on the computation of the next stage. For example, the backward propagation B1 of the first micro-batch of data on computing device C (corresponding to stage C) can only begin after the B1 computation on computing device D (corresponding to stage D) is completed.

[0037] Due to this dependency, each computing device will be idle while waiting for input data or gradients. For example, in the early stages of training, only computing device A starts processing F1, while computing devices B, C, and D are all waiting. This process results in a large number of wasted computation cycles, forming significant pipeline bubbles (e.g., ...). Figure 1A (As shown in the blank box in the middle), this phenomenon is particularly severe when there are many stages or few micro-batches, which restricts the utilization of hardware resources and training throughput.

[0038] To address these issues, interleaved pipeline parallelism has been proposed. By introducing virtual stages, computational tasks are further subdivided to achieve higher equipment utilization and fewer pipeline bubbles.

[0039] Figure 1B This is a schematic diagram of an interleaved pipeline parallel training process.

[0040] For example, such as Figure 1B As shown, the model is divided into multiple virtual stages, and multiple non-contiguous virtual stages are allocated to each computing device. Figure 1B Each computing device is assigned two virtual stages. For example, computing device A is assigned virtual stages A_1 and A_2, computing device B is assigned virtual stages B_1 and B_2, computing device C is assigned virtual stages C_1 and C_2, and computing device D is assigned virtual stages D_1 and D_2. The tasks of virtual stages A_1, B_1, C_1, and D_1 are indicated by white-marked squares, and the tasks of virtual stages A_2, B_2, C_2, and D_2 are indicated by gray-marked squares.

[0041] For example, in Figure 1B In the example, since each computing device is allocated two virtual stages, the total number of virtual stages is 2 × 4 = 8. Assuming the total number of layers in the model is 40, and using a uniform layer block method, each virtual stage contains 40 / 8 = 5 consecutive network layers. Due to the inherent properties of the interleaved pipeline parallel mode, each computing device will be allocated two non-consecutive layer blocks. In one example, computing device A performs training on layers 1-5 (virtual stage A_1) and layers 21-25 (virtual stage A_2); computing device B performs training on layers 6-10 (virtual stage B_1) and layers 26-30 (virtual stage B_2); computing device C performs training on layers 11-15 (virtual stage C_1) and layers 31-35 (virtual stage C_2); and computing device D performs training on layers 16-20 (virtual stage D_1) and layers 36-40 (virtual stage D_2).

[0042] Each computing device alternately executes the tasks of its assigned two virtual phases. For example, computing device A first processes the forward propagation of four micro-batches in virtual phase A_1 (indicated by white-marked squares F1~F4), then processes the forward propagation of four micro-batches in virtual phase A_2 (indicated by gray-marked squares F1~F4), then processes the forward propagation of two micro-batches in virtual phase A_1 (indicated by white-marked squares F5~F6), and then enters a waiting state. Subsequent processes are not described further. This "interleaved" execution method makes the tasks more intensive in the time dimension, reducing the waiting time of the computing devices.

[0043] In traditional non-interleaved pipelines, each computing device processes only one continuous stage, resulting in low resource utilization. However, by introducing virtual stages, a single computing device can handle multiple discontinuous virtual stages, increasing task density. Different computing devices can interleave forward and backward computations in the time dimension, reducing pipeline bubbles. Furthermore, the allocation of virtual stages can be dynamically adjusted based on the computational load of each stage, preventing overload of certain computing devices.

[0044] However, in Figure 1B In related technologies, such as those exemplified by [example technology], interleaved pipelined parallelism can only be executed when the number of layers handled by a single computing device (e.g., 10) is divisible by the number of virtual stages (e.g., 2). Here, the number of virtual stages refers to the number of virtual stages allocated to a single computing device in an interleaved pipelined parallelism strategy. Otherwise, when the number of layers handled by a single computing device is not divisible by the number of virtual stages, a rollback is required to execute a non-interleaved pipelined parallelism training process, which is detrimental to improving training efficiency. For example, when [example technology] Figure 1BWhen a single computing device in the process is responsible for 11 layers and has 2 virtual stages, interleaved pipelined parallelism is not executed; instead, it executes something like... Figure 1A The non-interleaved pipeline parallel training process is shown.

[0045] In other words, currently common distributed frameworks in the industry only support evenly divisible interleaved pipelined parallel training processes. Assume the total number of layers in the model is L, the number of computing devices participating in parallel training (i.e., pipeline parallelism) is P, and each computing device is responsible for M layers (i.e., M = L / P). In an evenly divisible interleaved pipelined parallel training process, M must be divisible by the number of virtual stages (i.e., the result of dividing M by the number of virtual stages is an integer with no remainder), and each virtual stage must be responsible for the same number of layers. If the number of layers M handled by a single computing device is not divisible by the number of virtual stages, some virtual stages will contain non-integer layers or have misaligned boundaries, disrupting task scheduling logic. Therefore, mainstream distributed frameworks typically revert to non-interleaved pipelined parallelism, abandoning the throughput improvement advantages of interleaved parallelism. This limitation significantly affects the generalization ability of interleaved pipelined parallelism to applicable scenarios.

[0046] Therefore, the divisibility constraint between the number of model layers, pipeline parallelism, and the number of virtual stages has become one of the key bottlenecks hindering efficient training deployment. Some methods attempt to address this issue by introducing decision-making mechanisms based on runtime metrics, such as determining the number of real model layers to be included in each virtual stage based on metrics like pipeline bubble time and memory usage. However, these methods rely on runtime metrics, requiring complex and time-consuming search and analysis optimizations, increasing deployment costs. Furthermore, these methods still have coupling and boundaries with the availability constraints of interleaved scheduling; for example, when the total number of model layers is not divisible by the pipeline parallelism, or when the system is unavailable, the model structure still needs to be reconfigured.

[0047] This disclosure provides at least one embodiment of a model training method, a model training apparatus, an electronic device, and a storage medium.

[0048] The model training method provided in at least one embodiment of this disclosure includes: in response to obtaining the insertion layer configuration corresponding to the first model, determining insertion layer information according to the insertion layer configuration, wherein the insertion layer configuration includes parallel strategy information, and the insertion layer information includes the number of insertion layers, the type of insertion layer, and the position information of the insertion layer; updating the first model according to the insertion layer information to obtain a second model; and performing interleaved pipeline parallel training on the second model.

[0049] In at least one embodiment of the model training method disclosed herein, an insertion layer-driven model construction scheme is proposed. This scheme introduces an insertion layer to rewrite the model structure during the graph construction phase (i.e., the model construction phase), thereby decoupling it from runtime performance metrics, avoiding complex search and analysis, and ensuring the availability of interleaved pipeline scheduling during the graph construction phase. Furthermore, it can simultaneously support non-uniform and non-continuous stage mapping. This method overcomes the rigid limitations of traditional distributed frameworks on integer constraints, improves system flexibility and compatibility, supports high-performance training under different model structures and different parallel configurations, and improves training efficiency.

[0050] The model training method provided in at least one embodiment of this disclosure can be applied to a distributed training system. In the embodiments of this disclosure, the distributed training system is configured to perform distributed training of the model, that is, to distribute the training task across multiple computing devices for collaborative completion. For example, the distributed training system may include multiple computing nodes, and each computing node may include multiple computing devices.

[0051] For example, computing devices may include graphics processing units (GPUs), tensor processing units (TPUs), general-purpose graphics processing units (GPGPUs), deep learning processing units (DPUs), accelerated processing units (APUs), neural network processing units (NPUs), application-specific integrated circuits (ASICs), or field-programmable gate arrays (FPGAs), etc. Of course, the embodiments of this disclosure are not limited to these, and computing devices may also be any other type of device.

[0052] Figure 2 A flowchart illustrating a model training method provided in at least one embodiment of this disclosure.

[0053] For example, such as Figure 2 As shown, the model training method provided in at least one embodiment of this disclosure may include steps S101 to S103.

[0054] Step S101: In response to obtaining the insertion layer configuration corresponding to the first model, determine the insertion layer information according to the insertion layer configuration, wherein the insertion layer configuration includes parallel strategy information, and the insertion layer information includes the number of insertion layers, the type of insertion layer, and the position information of the insertion layer.

[0055] Step S102: Update the first model based on the insertion layer information to obtain the second model.

[0056] Step S103: Perform interleaved pipeline parallel training on the second model.

[0057] First, it should be noted that in this embodiment, "first model" refers to the model that is currently being described, such as the initial model or the model to be updated, and is not used to indicate that it is the first model among multiple models. "Second model" refers to the model obtained after updating the first model, and subsequent interleaved pipelined parallel training will be performed based on the second model.

[0058] For example, insertion layer configurations can take the form of configuration files, command-line arguments, or code. They can be obtained by reading configuration files, passing parameters via the command line, or making function calls. The system can parse the insertion layer configuration to obtain the configuration information it contains. For instance, insertion layer configurations can be set by the user to define insertion rules, or they can be generated based on hardware resource configuration information.

[0059] For example, the insertion layer configuration may include parallel strategy information. Parallel strategy information records the parallel computing scheme of the model in a distributed training environment, such as the parallel mode used, the corresponding degree of parallelism, and the basic structural information of the model.

[0060] For example, an insert layer refers to a network layer that needs to be inserted into the first model; that is, an additional network layer added on top of the first model. Insert layers do not participate in loss calculations or gradient updates, so that the target model being trained remains unchanged. Insert layer information describes the properties of the insert layer to support its automatic addition to the first model. This information may include, for example, the number of insert layers, the type of insert layer, and the location of the insert layer.

[0061] For example, the number of insertion layers indicates the number of network layers that need to be inserted into the first model, which can be calculated based on parallel strategy information.

[0062] For example, the insertion layer type indicates the type of network layer to be inserted into the first model. When inserting multiple network layers, the insertion layer type can specify the type of each network layer separately, or all inserted network layers can be uniformly specified as the same type. The insertion layer type can include at least one of the following: identity layer, copy layer, delay layer, or memory layer, and different types of insertion layers can be selected according to actual needs. For example, by default, the insertion layer type can be uniformly determined as identity layer.

[0063] This disclosure provides at least one embodiment of a variety of parameterizable insertion layer primitives for inserting specified network layers into a model structure. These insertion layers do not participate in loss calculation and gradient update, so that the target model being trained is not altered, including but not limited to:

[0064] (1) Identity placeholder layer: No transformation is performed on the input data during forward propagation, no new parameters are introduced, and the input gradient is equal to the output gradient during backward propagation (gradient pass-through).

[0065] For example: Define the forward propagation process without performing any processing.

[0066] class IdentityInsert(nn.Module):

[0067] def forward(self, x):

[0068] return x

[0069] (2) Copy layer: only performs copy operations.

[0070] For example: Define a forward propagation process that performs only one copy computation.

[0071] class CopyAlignInsert(nn.Module):

[0072] def forward(self, x):

[0073] y = x.clone()

[0074] return y

[0075] (3) Controlling time consumption layer: Lightweight operator loop to control millisecond-level delay.

[0076] Here's an example: Define a forward propagation process and insert the calculation y = y + 0 multiple times (100 times in this example) to implement a lightweight operator loop.

[0077] class DelayInsert(nn.Module):

[0078] def __init__(self, iters: int = 100):

[0079] super().__init__()

[0080] self.iters = iters

[0081] def forward(self, x):

[0082] y = x

[0083] for _ in range(self.iters):

[0084] y = y + 0

[0085] return y

[0086] (4) Control memory layer: simulate actual memory usage by temporarily allocating a buffer.

[0087] For example: During forward propagation, a new tensor is allocated, thus occupying additional GPU memory.

[0088] class MemoryInsert(nn.Module):

[0089] def __init__(self, extra_bytes: int):

[0090] super().__init__()

[0091] self.extra_bytes = extra_bytes

[0092] def forward(self, x):

[0093] _buf = x.new_empty(self.extra_bytes / / x.element_size())

[0094] It should be noted that the above-mentioned insertion layer types are merely examples, and more types may be included as needed. This disclosure does not limit this.

[0095] For example, insertion layer position information indicates the insertion position of a network layer that needs to be inserted into the first model. When multiple layers are inserted, the insertion layer position information can indicate the insertion position of each network layer in the first model. The insertion layer position information can be calculated based on the number of inserted layers and parallel strategy information.

[0096] In some examples, the insertion layer location information may include an insertion location index and a relative location marker. The insertion location index indicates the sequence number of the network layer (reference layer) in the first model that serves as a location reference; the relative location marker indicates the insertion direction relative to the reference layer (e.g., "before" or "after"). The combination of these two allows for precise specification of the insertion location, such as inserting a network layer "after layer 4" in the first model.

[0097] For example, the insertion layer position information can be represented as {idx: 5, side: before}, indicating that the network layer is inserted before the 5th layer of the first model (and after the 4th layer); or as {idx: 10, side: after}, indicating that the network layer is inserted after the 10th layer of the first model (and before the 11th layer). In the above examples, idx represents the insertion position index, and side represents the relative position marker.

[0098] In some examples, the insertion layer position information may only include the insertion position index without explicitly including the relative position marker. In this case, the network layer (i.e., the insertion layer) is inserted before the reference layer determined by the insertion position index by default. Alternatively, the network layer may be inserted after the reference layer by default. This can be set according to actual needs, and the embodiments of this disclosure do not impose any restrictions on this.

[0099] In the model training method provided in at least one embodiment of this disclosure, the parallel strategy information may include, for example, pipeline parallelism, the number of virtual stages, and the total number of layers of the first model.

[0100] For example, pipeline parallelism can refer to the number of computing devices used to partition and execute the model in parallel within a pipelined parallel training strategy. Specifically, in pipelined parallelism, the model is partitioned into multiple consecutive stages according to layers, with each stage assigned to an independent computing device for training. Pipeline parallelism is the total number of computing devices used for this partitioning. For instance, if a model is allocated to 4 GPUs for pipelined parallel training, the pipeline parallelism is 4.

[0101] For example, the number of virtual stages can refer to the number of virtual stages allocated to a single computing device in an interleaved pipelined parallel strategy. For instance, in an interleaved pipelined parallel strategy, the model is divided into more virtual stages than the number of computing devices, and these virtual stages are not sequential but allocated to each computing device in an interleaved manner. In this case, the "number of virtual stages" in at least one embodiment of this disclosure refers to how many virtual stages are allocated to a single computing device.

[0102] For example, the total number of layers in the first model refers to the total number of layers in the first model from input to output, which is also the depth of the first model.

[0103] For example, suppose the first model has a total of 40 layers (L=40), a pipeline parallelism of 4 (P=4), and 2 virtual stages (V=2) allocated to a single computing device. This means there are four computing devices participating in parallel computing (denoted as computing devices A to D), and the total number of virtual stages is P×V=2×4=8. When using a uniform layer partitioning method, each virtual stage contains L / (P×V)=40 / 8=5 consecutive network layers. Due to the inherent properties of the interleaved pipeline parallel mode, each computing device will be allocated two non-consecutive layer blocks. In one example, computing device A performs training on layers 1-5 (virtual stage A_1) and layers 21-25 (virtual stage A_2) of the first model; computing device B performs training on layers 6-10 (virtual stage B_1) and layers 26-30 (virtual stage B_2); computing device C performs training on layers 11-15 (virtual stage C_1) and layers 31-35 (virtual stage C_2); and computing device D performs training on layers 16-20 (virtual stage D_1) and layers 36-40 (virtual stage D_2).

[0104] In the example above, since the number of layers handled by a single computing device (i.e., the total number of layers in the first model divided by the pipeline parallelism) is an integer multiple of the number of virtual stages, that is, the number of layers handled by a single computing device can be divided by the number of virtual stages, it is considered to satisfy the divisibility property. Even without inserting additional network layers, mainstream distributed frameworks can be reused for interleaved pipeline parallel training.

[0105] However, when the number of layers handled by a single computing device is not an integer multiple of the number of virtual stages, additional network layers need to be inserted to reuse mainstream distributed frameworks for interleaved pipelined parallel training. At least one embodiment of this disclosure provides a model training method that supports the automatic insertion of additional network layers into the model to meet the requirements of interleaved pipelined parallel training.

[0106] It should be noted that, in addition to the parallel strategy information, the insertion layer configuration provided in at least one embodiment of this disclosure may also include other information, which will be described in detail later.

[0107] In the model training method provided in at least one embodiment of this disclosure, an example of step S101 may include the following steps S201 to S202.

[0108] Step S201: Determine the number of insertion layers based on pipeline parallelism, number of virtual stages, and total number of layers in the first model.

[0109] Step S202: In response to the fact that the number of insertion layers is not equal to 0, determine the insertion layer position information based on the total number of layers in the first model and the number of insertion layers.

[0110] For example, in step S201, the number of insertion layers can be determined based on the pipeline parallelism, the number of virtual stages, and the total number of layers in the first model, so that the number of model layers after the insertion of network layers satisfies the divisibility property, that is, the value of [(number of insertion layers + total number of layers in the first model) / pipeline parallelism] can be divided by the number of virtual stages. In other words, the value of [(number of insertion layers + total number of layers in the first model) / pipeline parallelism] is an integer multiple of the number of virtual stages.

[0111] For example, one example of step S201 may include: determining the number of insertion layers based on the product of pipeline parallelism and the number of virtual stages, and the total number of layers in the first model.

[0112] In some examples, the number of insertion layers can be determined using the following formula:

[0113] D = ((P × V) - (L % (P × V))) % (P × V)

[0114] Where D represents the number of insertion layers, L represents the total number of layers in the first model, P represents the pipeline parallelism, and V represents the number of virtual stages.

[0115] Here is an example: When the total number of layers in the first model is 40, the pipeline parallelism is 4, and the number of virtual stages is 3, the calculated value is D = ((P × V) - (L % (P × V))) % (P × V) = (12-40%12)%12 = 8. That is, 8 additional network layers need to be inserted into the first model to satisfy the divisibility property.

[0116] In the model training method provided by at least one embodiment of this disclosure, the number of insertion layers can be automatically calculated, reducing the complexity of manual operations and improving the efficiency of the model training process. Furthermore, the number of insertion layers is calculated based on model structure information and parallel configuration information, providing support for high-performance training under different model structures and parallel configurations.

[0117] An example of step S101 may also include step S203.

[0118] Step S203: In response to the number of insertion layers being equal to 0, determine that the first model does not require insertion layers.

[0119] For example, when the calculated number of insertion layers is 0, it indicates that the divisibility property has been satisfied. Therefore, there is no need to insert additional network layers (i.e., insertion layers) into the first model, and the first model can be directly used for subsequent interleaved pipeline parallel training. Correspondingly, the determined insertion layer position information can be an empty list.

[0120] Of course, additional network layers may be inserted into the first model for other optimization purposes. This disclosure does not limit this, that is, "no need to insert" is not the same as "no insertion".

[0121] For example, one example of "determining the insertion layer location information based on the total number of layers and the number of insertion layers in the first model" in step S202 may include steps S2021 to S2022.

[0122] Step S2021: Determine the insertion interval value based on the total number of layers and the number of insertion layers in the first model.

[0123] Step S2022: Determine the first number of insertion position indices based on the insertion interval value to obtain the insertion layer position information, wherein the first number is equal to the number of insertion layers.

[0124] For example, in step S2021, if the number of insertion layers is not equal to 0, the insertion interval value can be calculated according to the following formula:

[0125]

[0126] Where G represents the insertion interval, D represents the number of insertion layers, and L represents the total number of layers in the first model. This indicates rounding down to the nearest integer.

[0127] It should be noted that the above formula is only an example, and other equivalent formulas can be used to calculate the insertion interval value according to actual needs.

[0128] For example, in step S2022, a loop can be used to calculate the insertion position index based on the insertion interval value, and the value of the insertion position index obtained in each loop can be stored in the index list. When the loop ends, the index list is determined as the insertion layer position information.

[0129] For example, in step S2022, after determining the first number of insertion position indices based on the insertion interval value, the first number of insertion position indices can be sorted in ascending order to obtain the insertion layer position information. Of course, the first number of insertion position indices can also be sorted in descending order, which can be set according to actual needs, and this embodiment of the disclosure does not limit this.

[0130] In the model training method provided by at least one embodiment of this disclosure, the insertion position index can be automatically calculated, reducing the complexity of manual operations and improving the efficiency of the model training process. Furthermore, the insertion position index is calculated based on model structure information, enhancing adaptability and generalization to different model architectures.

[0131] Figure 3 This is a flowchart illustrating a model training method provided in at least one embodiment of the present disclosure. For example, Figure 3 This is a specific example of steps S202 to S203 above.

[0132] For example, such as Figure 3 As shown, this example includes steps S301 to S309.

[0133] Step S301: Determine if the number of insertion layers D is 0. If the number of insertion layers D is equal to 0, proceed to step S302; if the number of insertion layers D is not equal to 0, proceed to step S303.

[0134] Step S302: Determine that the first model does not require an insertion layer, and determine that the insertion layer position information is an empty list.

[0135] Step S303: Determine the insertion interval value G based on the total number of layers L and the number of insertion layers D of the first model, where .

[0136] Step S304: Initialize the index list pList to an empty list, and initialize the insertion position index idx to the insertion interval value G.

[0137] Step S305: Determine if the following conditions are met: the length of the index list len(pList) is less than the number of insertion layers D, and the insertion position index idx is less than the total number of layers L of the first model. If met, proceed to step S306; otherwise, proceed to step S307.

[0138] Step S306: Store the value of the insertion position index idx into the index list pList, update the insertion position index idx according to idx=idx+G, and return to step S305.

[0139] Step S307: Determine if the following conditions are met: the length of the index list len(pList) is less than the number of insertion layers D, and the total number of layers L of the first model is greater than 0. If met, proceed to step S308; otherwise, proceed to step S309.

[0140] Step S308: Store the total number of layers L of the first model into the index list pList, and return to step S307.

[0141] Step S309: Determine the sorted index list pList as the insertion layer position information.

[0142] For example, suppose the total number of layers in the first model is L=40, and the number of insertion layers is D=8. Step S301 determines that the number of insertion layers is not equal to 0; then proceed to step S303 to calculate the insertion interval value. ; Step S304 is executed to initialize the index list pList = [], and the insertion position index idx = G = 5; Step S305 is executed. Since len(pList) = 0 < D and idx = 5 < L, the first loop is entered. Step S306 is executed to store the value 5 of idx into the index list pList (at this time, pList = [5]), and idx is updated to 10 according to idx = idx + G, then return to Step S305; Step S305 is executed. Since len(pList) = 1 < D and idx = 10 < L, the second loop is entered. Step S306 is executed to store the value 10 of idx into the index list pList (at this time, pList = [5, 10]), and idx is updated to 15 according to idx = idx + G, then return to Step S305. And so on. After the seventh loop is executed, pList = [5, 10, 15, 20, 25, 30, 35], idx = 40, and return to Step S305; Step S305 is executed. Since len(pList) = 7 < D and idx = 40 = L, the condition of Step S305 is not satisfied at this time, and jump to Step S307; Step S307 is executed. Since len(pList) = 7 < D and L = 40 > 0, the condition of Step S307 is satisfied, and jump to Step S308; Step S308 is executed to store the value 40 of L into the index list pList. At this time, pList = [5, 10, 15, 20, 25, 30, 35, 40], and return to Step S307; Step S307 is executed. Since len(pList) = 8 = D and L = 40 > 0, the condition of Step S307 is not satisfied, and jump to Step S309. The index list pList = [5, 10, 15, 20, 25, 30, 35, 40] sorted from small to large is determined as the insertion layer position information (the sorting step can be omitted here).

[0143] In the above example, it can be defaulted that the network layer (i.e., the insertion layer) is inserted before the reference layer determined by the insertion position index. For example, in the above example, a total of 8 insertion layers need to be inserted into the first model, and they are inserted before the 5th layer, 10th layer, 15th layer, 20th layer, 25th layer, 30th layer, 35th layer, and 40th layer of the first model respectively. It should be noted that it can also be defaulted that the network layer is inserted after the reference layer, which can be set according to actual needs, and the embodiments of the present disclosure do not limit this.

[0144] In the model training method provided by at least one embodiment of the present disclosure, the insertion layer configuration may further include at least one of the following: insertion layer enable configuration, insertion layer type configuration, insertion layer position configuration, or custom insertion layer configuration.

[0145] For example, the insertion layer enable configuration is used to determine whether to enable the insertion strategy. For instance, when the insertion layer enable configuration is True, the insertion strategy is enabled; when the insertion layer enable configuration is False, the insertion strategy is disabled. An example of an insertion layer enable configuration is `enable: true`, which ensures that the insertion strategy is enabled.

[0146] For example, one example of step S101 may also include step S204.

[0147] Step S204: In response to the insertion layer configuration including the insertion layer type configuration, determine the insertion layer type based on the insertion layer type configuration.

[0148] In step S204, the insertion layer configuration can explicitly include the insertion layer type configuration. In this case, the insertion layer type can be determined primarily based on the insertion layer type configuration. An example of the insertion layer type configuration is insert_type: { type:copy_align}, in which case the insertion layer type can be determined to be a copy layer.

[0149] For example, when the insertion layer configuration does not include an insertion layer type configuration, the insertion layer type can be defaulted to an identical placeholder layer. Of course, it can also be defaulted to other types, and this embodiment of the disclosure does not limit this.

[0150] For example, one example of step S101 may also include step S205.

[0151] Step S205: In response to the insertion layer configuration including the insertion layer position configuration, and the insertion layer position configuration conforming to the first preset rule, determine the insertion layer position information according to the insertion layer position configuration.

[0152] In step S205, the insertion layer configuration may explicitly include the insertion layer position configuration. If the insertion layer position configuration is determined to conform to the first preset rule, the insertion layer position information can be determined preferentially based on the insertion layer position configuration. An example of the insertion layer position configuration is positions: [3, 7, 11, 15, 19, 23, 27, 31]. It is possible to default that the network layer is inserted before the reference layer determined by the insertion layer position configuration. In this case, it can be determined that the network layer is inserted before layers 3, 7, 11, 15, 19, 23, 27, and 31 of the first model. That is, all insertion position indices in the insertion layer position configuration can be used as insertion position indices in the insertion layer position information, and the relative position marker in the insertion layer position information is set to "before" by default. It should be noted that the network layer can also be inserted after the reference layer by default (that is, the relative position marker in the insertion layer position information is set to "after" by default). This can be set according to actual needs, and this embodiment does not limit this.

[0153] For example, a validity check can be performed to detect whether the insertion layer position configuration conforms to a first preset rule. The first preset rule may include, for example, that the length of the insertion layer position configuration is equal to the number of insertion layers, that all insertion position indices in the insertion layer position configuration are within the valid range of the first model (e.g., the value of the insertion position index does not exceed the total number of layers in the first model), and that each insertion position index is unique.

[0154] For example, in step S205, if it is determined that the insertion layer position configuration conforms to the first preset rule, the insertion position indexes in the insertion layer position configuration can be sorted in ascending order before determining the insertion layer position information. Of course, the insertion position indexes in the insertion layer position configuration can also be sorted in descending order, which can be set according to actual needs, and this embodiment does not limit this.

[0155] In some examples, when the insertion layer position configuration does not conform to the first preset rule, or when the insertion layer configuration does not include the insertion layer position configuration, the insertion layer position information can be automatically determined according to the above step S202.

[0156] In the model training method provided by at least one embodiment of this disclosure, by setting the insertion layer position configuration, the model structure can be adjusted more flexibly to adapt to diverse training needs and task scenarios. Simultaneously, the stability of the model structure adjustment process is improved through verification using a first preset rule.

[0157] For example, after step S101 and before step S102, the model training method provided in at least one embodiment of this disclosure may further include the following step S104.

[0158] Step S104: In response to the insertion layer configuration including a custom insertion layer configuration, and the custom insertion layer configuration conforming to the second preset rule, update the insertion layer information based on the custom insertion layer configuration.

[0159] In step S104, the custom insertion layer configuration can specify the insertion layer type and insertion layer position information. An example of a custom insertion layer configuration is as follows:

[0160] insertions:

[0161] - { idx: 1, before: { type: copy_align}}

[0162] - { idx: 4, after: { type: delay, iters: 100}}

[0163] Based on the example above, two custom insertion layers can be identified. The first custom insertion layer is a copy layer, inserted before the first layer of the first model; the second custom insertion layer is a time-controlled layer, with a loop count of 100, inserted after the fourth layer of the first model.

[0164] In some examples, the priority of a custom insertion layer is higher than that of the insertion layers determined according to the above embodiments, meaning it can override the insertion layers determined in the above embodiments. For example, after determining the relevant information of the custom insertion layer, the previously determined insertion layer information can be updated using the relevant information of the custom insertion layer. For example, processing the information in the custom insertion layer configuration one by one, when the index idx is the same as the insertion position index of a certain entry in the previously determined insertion layer information, the information in the custom insertion layer configuration can be used to overwrite the corresponding entry in the insertion layer information; when the index idx is different from the insertion position index of any entry in the previously determined insertion layer information, the update operation is skipped and an alarm is triggered to indicate that there may be an anomaly in the custom insertion configuration. That is, the above second preset rule is that the index idx in the custom insertion layer configuration is the same as the insertion position index of a certain entry in the previously determined insertion layer information, that is, the index idx is located in the determined insertion layer information.

[0165] After step S104, the entries in the insertion layer information can be sorted in ascending order of insertion position index. Of course, the entries in the insertion layer information can also be sorted in descending order of insertion position index, which can be set according to actual needs. This embodiment of the present disclosure does not limit this.

[0166] In some examples, when the insertion layer configuration does not include a custom insertion layer configuration and no sorting operation has been performed in the previous steps, the entries in the insertion layer information can be sorted in ascending order of insertion position index. Of course, the entries in the insertion layer information can also be sorted in descending order of insertion position index, depending on actual needs. This disclosure does not impose any restrictions on this.

[0167] In the model training method provided by at least one embodiment of this disclosure, by setting a custom insertion layer configuration, the model structure can be adjusted more flexibly to adapt to diverse training needs and task scenarios. Simultaneously, the stability of the model structure adjustment process is improved through verification using a second preset rule.

[0168] The following is an example of an insertion layer configuration provided in at least one embodiment of the present disclosure.

[0169] enable: true / / Insertion layer enable configuration

[0170] P: 4 / / Pipeline parallelism

[0171] V: 3 / / Number of virtual stages

[0172] num_block: 40 / / Total number of layers in the first model

[0173] insert_type: { type: identity} / / Insertion layer type configuration

[0174] positions: [3, 7, 11, 15, 19, 23, 27, 31] / / Insertion layer position configuration

[0175] insertions: / / Custom insertion layer configuration

[0176] - { idx: 0, before: { type: copy_align}}

[0177] - { idx: 4, after: { type: delay, iters: 100}}

[0178] In the model training method provided in at least one embodiment of this disclosure, the insertion layer information can be represented in dictionary form. For example, a data structure containing the following fields can be defined: insertion position index, relative position marker, and insertion layer type.

[0179] class BlockAction:

[0180] idx: int # Insertion position index

[0181] side: str # Relative position marker

[0182] spec: dict # Insertion level type

[0183] Example fields are as follows: [

[0185] BlockAction(idx=5, side='before', spec={'type': 'identity'}),

[0186] BlockAction(idx=10, side='before', spec={'type': 'identity'}) ]

[0188] In the example above, the number of insertion layers is 2. The first insertion layer is an identity placeholder layer, inserted before the 5th layer of the first model; the second insertion layer is also an identity placeholder layer, inserted before the 10th layer of the first model.

[0189] In the model training method provided by at least one embodiment of the present disclosure, the attribute information of each insertion layer can be flexibly described and managed in the form of a dictionary, so that the insertion operation has good scalability and configurability, and facilitates the structural adjustment of the model.

[0190] Figure 4 This is a flowchart illustrating a model training method provided in at least one embodiment of the present disclosure. For example, Figure 4 This is a specific example of steps S101 and S104 above.

[0191] Step S401: Determine whether the insertion layer configuration has been obtained. If yes, proceed to step S402; otherwise, proceed to step S414.

[0192] Step S402: Calculate the number of insertion layers. For example, refer to step S201.

[0193] Step S403: Determine whether an insertion layer position configuration exists and is valid. If yes, proceed to step S404; otherwise, proceed to step S405. For example, step S403 can be understood as determining whether the following conditions are met: the insertion layer configuration includes an insertion layer position configuration, and the insertion layer position configuration conforms to the first preset rule.

[0194] Step S404: Determine the insertion layer position information based on the insertion layer position configuration. For example, refer to step S205.

[0195] Step S405: Determine the insertion layer position information based on the total number of layers and the number of insertion layers in the first model. For example, refer to step S202.

[0196] Step S406: Determine the insertion layer information. For example, step S406 can be understood as integrating the number of insertion layers, the type of insertion layer, and the position information of the insertion layer to obtain complete insertion layer information. The step of obtaining the insertion layer type is omitted in this flowchart; please refer to the aforementioned embodiment.

[0197] Step S407: Determine whether a custom insert layer configuration exists. If yes, proceed to step S409; otherwise, proceed to step S408. For example, step S407 can be understood as determining whether the insert layer configuration includes a custom insert layer configuration.

[0198] Step S408: Sort the entries in the insertion layer information. For example, step S408 can be understood as sorting the entries in the insertion layer information in ascending or descending order of their insertion position index.

[0199] Step S409: Process the information in the custom insertion layer configuration one by one.

[0200] Step S410: Determine whether the index is located in the insertion layer information. If yes, proceed to step S411; otherwise, proceed to step S412. For example, step S410 can be understood as determining whether the index in the custom insertion layer configuration is the same as the insertion position index of a certain entry in the previously determined insertion layer information.

[0201] Step S411: Update the insertion layer information based on the custom insertion layer configuration. For example, refer to step S104.

[0202] Step S412: Skip the update and issue an alert.

[0203] Step S413: Sort the entries in the insertion layer information. For example, step S413 can be understood as sorting the entries in the insertion layer information in ascending or descending order of their insertion position index.

[0204] Step S414: Determine not to insert a layer in the first model.

[0205] The model training method provided in at least one embodiment of this disclosure supports multi-level structured configuration, which can flexibly adjust the model structure to adapt to diverse training needs and task scenarios, thereby improving the scalability and deployment flexibility of the model.

[0206] In the model training method provided in at least one embodiment of this disclosure, an example of step S102 may include the following steps S211 to S212.

[0207] Step S211: Generate the corresponding insertion layer based on the insertion layer information.

[0208] Step S212: Insert the insertion layer into the specified position in the first model. The specified position is determined based on the insertion layer position information.

[0209] For example, in step S211, the data in the insertion layer information (e.g., in the form of a dictionary) can be traversed, the insertion layer position information (insertion position index and relative position mark) and the insertion layer type can be read item by item, and the corresponding insertion module can be instantiated.

[0210] For example, the following is an example of instantiating an insertion module, which is equivalent to generating an identity placeholder layer:

[0211] insert_module = IdentityInsert()

[0212] For example, in step S212, the Insert function or other equivalent model modification interface can be called to insert the instantiated insertion module (i.e., the insertion layer) into a specified position in the model, thereby modifying the model structure. If the insertion layer information involves multiple insertion layers, the Insert function can be called repeatedly, with the call to the Insert function executed synchronously during the traversal of the insertion layer information. For example, in each loop, the input parameters of the Insert function can be the current model (initially the first model), the instantiated insertion module, and the target insertion position index, and the output parameter is the updated current model (as the input parameter for the next loop). The instantiated insertion module is determined by the type of the currently traversed insertion layer. The target insertion position index is determined by the currently traversed insertion position index and the relative position marker. If the relative position marker is "before", then the target insertion position index = the insertion position index; if the relative position marker is "after", then the target insertion position index = the insertion position index + 1. It should be noted that since each insertion operation increases the number of model layers, the following scheme is adopted in this embodiment to avoid insertion misalignment: when the entries in the insertion layer information are arranged in descending order of insertion position index, sequential traversal is used when traversing the insertion layer information; when the entries in the insertion layer information are arranged in ascending order of insertion position index, reverse traversal is used when traversing the insertion layer information. Both of these schemes ensure that the network layers are inserted into the model sequentially from back to front (from large index to small index), thereby ensuring that each insertion operation does not change the index and relative position of the layers before the inserted layer, effectively avoiding coordinate offset problems caused by insertion.

[0213] In the model training method provided by at least one embodiment of this disclosure, different types of network layers can be inserted at different positions in the model, effectively improving the flexibility and scalability of the model architecture. By inserting in reverse order, insertion misalignment caused by changes in the number of layers is avoided, ensuring the accuracy and reliability of the insertion operation.

[0214] The second model obtained by the model training method provided in at least one embodiment of this disclosure can pass the algebraic consistency check and schedulability check, thereby meeting the requirements of interleaved pipeline parallel training. The algebraic consistency check and schedulability check will be described below.

[0215] It's worth noting here that, similar to the interpolation layer configuration, the logical allocation scheme for each layer of the first model can also be specified through configuration files or command-line parameters, or generated based on a preset algorithm (e.g., equal partitioning). The logical allocation scheme refers to the organization and mapping strategy for each layer in the first model without introducing an interpolation layer, reflected in the device partitioning and virtual stage partitioning under a pipelined parallel architecture. For example, the logical allocation scheme can be described by the following logical vectors A and B.

[0216] Logical vector A: Describes the number of logical layers (excluding insertion layers) carried by each actual pipeline node (computing device). For example, if the first model has 40 layers and the pipeline parallelism is 4, A=[8,12,11,9] means that computing device 1 carries 8 network layers of the first model, computing device 2 carries 12 network layers of the first model, computing device 3 carries 11 network layers of the first model, and computing device 4 carries 9 network layers of the first model.

[0217] Logical vector B: Describes the number of logical layers (excluding insertion layers) carried by each virtual stage. For example, if the first model has 40 layers and a pipeline parallelism of 4, and the number of virtual stages allocated to a single computing device is 3, then B=[2,2,4,4,4,4,4,4,3,2,3,4] means that virtual stage 1 carries 2 network layers of the first model, virtual stage 2 carries 2 network layers of the first model, ..., virtual stage 12 carries 4 network layers of the first model.

[0218] Since the second model has been obtained through the aforementioned step S102, based on the network layer distribution of the second model, the above logical vectors can be reconstructed to obtain the following physical vectors:

[0219] Physical vector A': Describes the number of physical layers (including inserted layers) carried by each real pipeline node (computing device). For example, the first model has 40 layers and a pipeline parallelism of 4. After inserting network layers into the first model, the physical vector A' corresponding to the second model is [12,12,12,12], indicating that computing device 1 carries 12 network layers of the second model, computing device 2 carries 12 network layers of the second model, computing device 3 carries 12 network layers of the second model, and computing device 4 carries 12 network layers of the second model.

[0220] Physical vector B': Describes the number of physical layers (including insertion layers) carried by each virtual stage. For example, the first model has 40 layers, a pipeline parallelism of 4, and 3 virtual stages allocated to a single computing device. The physical vector B' corresponding to the second model is [4,4,4, 4,4,4, 4,4,4, 4,4,4].

[0221] The following sections will describe algebraic consistency checks and schedulability checks.

[0222] 1. Algebraic consistency check

[0223] The algebraic consistency check is considered passed if and only if all three of the following constraints are met simultaneously.

[0224] (1) Length constraint

[0225] len(A')=P, len(B')=P×V, where len(A') represents the length of physical vector A' and len(B') represents the length of physical vector B'.

[0226] (2) Summation Consistency Constraint

[0227] sum(A')=sum(B')=L', where sum(A') represents the summation of the elements in the physical vector A', sum(B') represents the summation of the elements in the physical vector B', and L' is the total number of physical layers, which is the total number of layers in the first model plus the number of inserted layers, which is also the total number of layers in the second model.

[0228] (3) Segmentation and Consistency Constraints

[0229] When the physical vector B' is divided into P consecutive segments by step size V, sum(B'[i×V : (i+1)×V]) = A'[i], that is, the sum of the elements in each segment is equal to the corresponding elements in the physical vector A'.

[0230] In one example, P=4, V=3, L=40, A=[8,12,11,9], B=[2,2,4,4,4,4,4,4,3,2,3,4]. If any network layer is inserted into the first model, the physical vector B'=[4,4,4, 4,4,4, 4,4,4, 4,4,4,4] is obtained. At this time, len(B')=15, P×V=12, so len(B')≠P×V, which does not satisfy the length constraint.

[0231] However, according to the model training method provided in at least one embodiment of this disclosure, the total number of layers of the second model can be obtained as L'=48, and the physical segmentation granularity can be further set as k=L' / (P×V)=4, which can simulate the network layer distribution of the second model. On this basis, the physical vectors can be reconstructed from the logical vectors: A'=[12,12,12,12], B'=[4,4,4,4,4,4,4,4,4,4,4,4,4]. At this point, it is determined whether the physical vector satisfies the above three constraints: len(A')=4=P, len(B')=12=P×V, which satisfies the length constraint; sum(A')=48=L', sum(B')=48=L', which satisfies the summation consistency constraint; sum(B'[0:3]) = 4+4+4 = 12 = A'[0], sum(B'[3:6]) = 4+4+4 = 12 = A'[1], sum(B'[6:9]) = 4+4+4 = 12=A'[2], sum(B'[9:12]) = 4+4+4 = 12 = A'[3], which satisfies the segmentation and consistency constraints.

[0232] Since the physical vectors satisfy the above three constraints, the algebraic consistency check is passed.

[0233] 2. Scheduling Verification

[0234] First, it should be noted that the conditions for schedulability verification need to be combined with the hard constraints set by the scheduler (e.g., non-interleaved 1F1B, interleaved 1F1B, etc.). The following example uses the interleaved 1F1B pipeline parallel mode:

[0235] When performing interleaved 1F1B pipelined parallel training, it is typically required that the number of physical layers carried by each virtual stage be constant, and that L'% (P×V) = 0 (or an equivalent condition). If these conditions are not met, an error message will be displayed prompting the user to adjust the configuration or automatically degrade to non-interleaved 1F1B pipelined parallel mode for training. If the above verification conditions are met, the second model can then be divided into P×V layers of length k with a physical partitioning granularity of k=L' / (P×V), and assigned to each virtual stage.

[0236] For example, if no additional network layers are inserted, then L'=L=40, P=4, V=3, then L'% (P×V)= 40 % 12 ≠ 0. Since it is impossible to divide 40 layers evenly into 12 equal-length blocks, the schedulability check condition of interleaved scheduling is not met.

[0237] The role of the insertion layer is to transform the aforementioned non-uniform logical structure into a uniform physical scheduling structure. For example, if L'=48, P=4, and V=3, then L'% (P×V)= 48 % 12 = 0. Since the 48 layers can be uniformly divided into 12 equal-length blocks, the schedulability check condition of the interleaved scheduling is satisfied.

[0238] In the model training method provided in at least one embodiment of this disclosure, an example of step S103 may include: dividing the second model into multiple layers according to the number of virtual stages and the pipeline parallelism, and mapping the multiple layers to corresponding computing devices for interleaved pipeline parallel training.

[0239] For example, the second model can be divided into P×V layers of length k with a physical partitioning granularity of k=L' / (P×V). These layers are then assigned to various virtual stages and mapped to corresponding computing devices for interleaved pipelined parallel training. Here, L' is the total number of physical layers, which is the total number of layers in the first model plus the number of inserted layers, and also the total number of layers in the second model.

[0240] For example, suppose the second model has a total of 40 layers (L'=40), a pipeline parallelism of 4 (P=4), and 2 virtual stages allocated to a single computing device (V=2). This means there are four computing devices participating in parallel computing (denoted as computing device 1 to computing device 4), and the total number of virtual stages is P×V=2×4=8. The second model is then divided into 8 equal blocks of length 5 according to a physical partitioning granularity of k=L' / (P×V)=40 / 8=5, and these blocks are allocated to each virtual stage. Each virtual stage contains 5 network layers. Due to the inherent properties of the interleaved pipeline parallelism mode, each computing device will be allocated two non-contiguous blocks. In one example, computing device 1 performs training on layers 1-5 (virtual phase 1) and layers 21-25 (virtual phase 2) of the second model; computing device 2 performs training on layers 6-10 (virtual phase 3) and layers 26-30 (virtual phase 4); computing device 3 performs training on layers 11-15 (virtual phase 5) and layers 31-35 (virtual phase 6); and computing device 4 performs training on layers 16-20 (virtual phase 7) and layers 36-40 (virtual phase 8). That is, the physical layers executed by the computing devices are discontinuous, a result of interleaved scheduling.

[0241] The model training method provided in at least one embodiment of the present disclosure supports non-uniform logic layer allocation and non-continuous physical layer execution, thereby improving pipeline utilization while taking into account the flexibility of model structure adjustment.

[0242] The following is an example of a model training method provided by at least one embodiment of the present disclosure.

[0243] For example, the first model has 40 layers, a pipeline parallelism of 4, and 3 virtual stages allocated to a single computing device. The resulting insertion layer configuration is as follows:

[0244] enable: true / / Insertion layer enable configuration

[0245] P: 4 / / Pipeline parallelism

[0246] V: 3 / / Number of virtual stages

[0247] num_block: 40 / / Total number of layers in the first model

[0248] First, calculate the number of insertion layers D: D = ((P × V) - (L % (P × V))) % (P × V) = (12-40%12)%12 = 8, meaning that 8 additional network layers need to be inserted into the first model. Next, since the insertion layer configuration does not include insertion layer position configuration, the insertion layer position information is determined based on the total number of layers in the first model and the number of insertion layers D. Steps S301 to S309 are executed to obtain the index list pList=[5,10,15,20,25,30,35,40], which is used as the insertion position index in the insertion layer position information, and the default relative position is marked as before. Since the insertion layer configuration does not include insertion layer type configuration, the insertion layer type is determined to be an identical placeholder layer by default. Based on the determined number of insertion layers, insertion layer type, and insertion layer position information, the insertion layer information is as follows: [

[0250] BlockAction(idx=5, side='before', spec={'type': 'identity'}),

[0251] BlockAction(idx=10, side='before', spec={'type': 'identity'}),

[0252] BlockAction(idx=15, side='before', spec={'type': 'identity'}),

[0253] BlockAction(idx=20, side='before', spec={'type': 'identity'}),

[0254] BlockAction(idx=25, side='before', spec={'type': 'identity'}),

[0255] BlockAction(idx=30, side='before', spec={'type': 'identity'}),

[0256] BlockAction(idx=35, side='before', spec={'type': 'identity'}),

[0257] BlockAction(idx=40, side='before', spec={'type': 'identity'}) ]

[0259] After obtaining the insertion layer information, step S102 is executed to traverse the insertion layer information, instantiate the insertion module, and call the insertion function to insert the insertion module into the first model to obtain the second model.

[0260] After obtaining the second model, step S103 is executed, in which the second model can be divided into P×V=12 layers of length 4 according to the physical partitioning granularity of k=L' / (P×V)=4. The second model is divided into P×V=12 layers of length 4 and assigned to each virtual stage. The layers are mapped to the corresponding computing devices for interleaved pipeline parallel training and can reuse mainstream distributed frameworks.

[0261] In some examples, to facilitate flexible switching of training strategies or external distribution of model weights, a "rollback save" strategy can be adopted, that is, after the second model is trained, the inserted layer in the second model is deleted, and then the trained model weights are mapped and saved according to the layer index.

[0262] For example, after step S103, the model training method provided in at least one embodiment of this disclosure may further include steps S105 to S106.

[0263] Step S105: Based on the insertion layer information, perform the insertion layer deletion operation on the trained second model to obtain the first model carrying weights.

[0264] Step S106: Save the weights of the first model.

[0265] In step S105, since the insertion layer is determined based on the insertion layer information, this information can be considered an insertion record. An insertion layer deletion operation can be performed on the second model based on this record, restoring the model structure to be consistent with the original model, i.e., obtaining the first model carrying the weights. One example of the insertion layer deletion operation is a reverse deletion operation, which specifically includes: arranging the network layers of the second model in descending order according to their layer indices, and sequentially determining whether each layer is an insertion layer. If so, removing it from the second model, until all insertion layers in the second model have been removed. This reverse deletion operation effectively avoids index offset problems caused by layer deletion, ensuring the accuracy of the model structure.

[0266] In step S106, the weights of the first model can be mapped and saved according to the layer index, so that the weights are aligned with the model structure, making it easier to load them into other models later.

[0267] The model training method provided in at least one embodiment of this disclosure supports flexible reconstruction of the model structure before training, and restores the original structure through a rollback mechanism during the weight saving stage, thereby ensuring the flexibility of the training process and the efficiency of pipeline parallelism while meeting the standardization requirements of model deployment.

[0268] After step S105, the training strategy can be changed. For example, step S101 can be re-executed to obtain a new insertion layer configuration, determine the new insertion layer information, and update the first model according to the new insertion layer information to obtain the third model. Subsequently, the third model can be trained in an interleaved pipeline parallel manner.

[0269] Through the above methods, the model training method provided in at least one embodiment of this disclosure supports flexible switching of training strategies, which can effectively improve training efficiency.

[0270] Figure 5 This is an exemplary schematic diagram of a weight saving method provided for at least one embodiment of the present disclosure.

[0271] For example, such as Figure 5 As shown, in response to triggering the model weight export, the following steps S501 to S509 are executed.

[0272] Step S501: Based on the insertion layer information, determine whether the second model has an insertion layer. If it does not exist, proceed to step S502; if it does exist, proceed to step S503.

[0273] Step S502: Map and save the weights of the second model according to the layer index.

[0274] Step S503: Arrange the network layers of the second model in descending order according to their layer indices.

[0275] Step S504: Based on the insertion layer information, determine whether the current layer is an insertion layer. If yes, proceed to step S505; otherwise, take the next layer as the current layer and continue with step S504.

[0276] Step S505: Remove the current layer from the second model.

[0277] Step S506: Determine whether all inserted layers in the second model have been removed. If yes, proceed to step S507; otherwise, return to step S504.

[0278] Step S507: Obtain the first model carrying weights, map and save the weights of the first model according to the layer index.

[0279] Step S508: Determine whether training needs to continue. If yes, proceed to step S509. If no, end the process.

[0280] Step S509: Change the training strategy.

[0281] For a detailed description of steps S501 to S509, please refer to the above description of steps S105 to S106, which will not be repeated here.

[0282] In other examples, loading pre-trained model weights is also supported to achieve transfer learning or incremental learning. For example, prior to step S103, the model training method provided in at least one embodiment of this disclosure may further include steps S107 to S109.

[0283] Step S107: Based on the insertion layer information, perform an insertion layer deletion operation on the second model to obtain the first model.

[0284] Step S108: Load the pre-stored weights that match the structure of the first model into the first model.

[0285] Step S109: Update the first model carrying weights according to the insertion layer information to obtain the second model carrying weights.

[0286] In step S107, based on the insertion layer information, the insertion layer deletion operation can be performed on the second model obtained according to step S102, temporarily rolling back its network structure and restoring it to the first model consistent with the original structure.

[0287] In step S108, pre-stored weights (i.e., the original model weights that have been trained) that match the structure of the first model can be loaded into the first model.

[0288] In step S109, based on the insertion layer information, the corresponding insertion layer can be re-inserted into the first model carrying weights, while keeping the original layer parameters unchanged, to obtain the second model carrying weights.

[0289] After completing the above weight loading and structure restoration process, the subsequent training process can be carried out based on the second model. The above method can improve the training flexibility and realize the transfer of pre-trained knowledge.

[0290] It should also be noted that the execution order of the various steps of the model training method in the various embodiments of this disclosure is not limited. Although the execution process of each step has been described in a specific order above, this does not constitute a limitation on the embodiments of this disclosure. The various steps in the model training method can be executed sequentially or in parallel, which can be determined according to actual needs.

[0291] For example, compared to the above description, the model training method provided in at least one embodiment of this disclosure may include more or fewer steps, and the embodiments of this disclosure do not limit this.

[0292] Figure 6 This is a schematic block diagram of a model training apparatus provided for at least one embodiment of the present disclosure. The model training apparatus may, for example, be a component of a distributed training system.

[0293] For example, such as Figure 6 As shown, the model training apparatus 600 provided in at least one embodiment of this disclosure may include a determination module 601, an update module 602, and a training module 603.

[0294] In some examples, the determination module 601 is configured to determine the insertion layer information based on the insertion layer configuration corresponding to the first model in response to obtaining the insertion layer configuration. The insertion layer configuration includes parallel strategy information, and the insertion layer information includes the number of insertion layers, the type of insertion layer, and the position information of the insertion layer.

[0295] In some examples, the update module 602 is configured to update the first model based on the insertion layer information to obtain the second model.

[0296] In some examples, training module 603 is configured to perform interleaved pipelined parallel training on the second model.

[0297] For example, in at least one embodiment of this disclosure, the parallel strategy information includes pipeline parallelism, the number of virtual stages, and the total number of layers in the first model. The determination module 601 includes a first determination unit and a second determination unit. The first determination unit is configured to determine the number of insertion layers based on the pipeline parallelism, the number of virtual stages, and the total number of layers in the first model. The second determination unit is configured to determine the insertion layer position information based on the total number of layers in the first model and the number of insertion layers in response to the insertion layer number not being equal to 0.

[0298] For example, in at least one embodiment of this disclosure, the first determining unit is further configured to determine the number of insert layers based on the product of pipeline parallelism and the number of virtual stages, and the total number of layers of the first model.

[0299] For example, in at least one embodiment of this disclosure, the second determining unit is further configured to determine an insertion interval value based on the total number of layers and the number of insertion layers of the first model; and to determine a first number of insertion position indices based on the insertion interval value to obtain insertion layer position information, wherein the first number is equal to the number of insertion layers.

[0300] For example, in at least one embodiment of this disclosure, the insertion layer configuration further includes at least one of the following: insertion layer enable configuration, insertion layer type configuration, insertion layer position configuration, or custom insertion layer configuration, wherein the insertion layer enable configuration is used to determine whether to enable the insertion strategy.

[0301] For example, in at least one embodiment of this disclosure, the determining module 601 further includes a third determining unit. The third determining unit is configured to determine an insert layer type based on the insert layer type configuration in response to the insert layer configuration including the insert layer type configuration.

[0302] For example, in at least one embodiment of this disclosure, the determining module 601 further includes a fourth determining unit. The fourth determining unit is configured to determine insertion layer position information based on the insertion layer position configuration in response to the insertion layer configuration including an insertion layer position configuration and the insertion layer position configuration conforming to a first preset rule.

[0303] For example, the model training apparatus 600 provided in at least one embodiment of this disclosure further includes a custom module. The custom module is configured to update the insertion layer information based on the custom insertion layer configuration in response to the insertion layer configuration including a custom insertion layer configuration and the custom insertion layer configuration conforming to a second preset rule.

[0304] For example, in at least one embodiment of this disclosure, the insertion layer type includes at least one of the following: an identity placeholder layer, a copy layer, a time-consuming control layer, or a memory control layer.

[0305] For example, in at least one embodiment of this disclosure, the insertion layer position information includes an insertion position index and a relative position marker.

[0306] For example, in at least one embodiment of this disclosure, the update module 602 includes a generation unit and an insertion unit. The generation unit is configured to generate a corresponding insertion layer based on the insertion layer information; the insertion unit is configured to insert the insertion layer into a specified position in the first model, the specified position being determined based on the insertion layer position information.

[0307] For example, in at least one embodiment of this disclosure, the training module 603 is further configured to divide the second model into multiple layers based on the number of virtual stages and the pipeline parallelism, and map the multiple layers to corresponding computing devices for interleaved pipeline parallel training.

[0308] For example, the model training apparatus 600 provided in at least one embodiment of this disclosure further includes a weight storage module. The weight storage module is configured to: perform an insertion layer deletion operation on the trained second model according to the insertion layer information to obtain a first model carrying weights; and store the weights of the first model.

[0309] For example, the model training apparatus 600 provided in at least one embodiment of this disclosure further includes a weight import module. The weight import module is configured to perform an insertion layer deletion operation on the second model according to the insertion layer information to obtain a first model; load pre-stored weights that match the structure of the first model into the first model; and update the first model carrying the weights according to the insertion layer information to obtain a second model carrying the weights.

[0310] It should be noted that the various modules and units described above can be implemented by software, hardware, firmware, or any combination thereof. For example, the determination module, update module, and training module can be implemented as a determination circuit, an update circuit, and a training circuit, respectively. The embodiments of this disclosure do not limit their specific implementation methods.

[0311] It should be understood that the model training apparatus 600 provided in at least one embodiment of this disclosure can be used to implement the aforementioned model training method and can also achieve similar technical effects as the aforementioned model training method, which will not be elaborated here.

[0312] It should be noted that, in the embodiments of this disclosure, the model training device 600 may include more or fewer modules or units, and the connection relationship between the various modules or units is not limited and can be determined according to actual needs. The specific configuration of each module or unit is not limited; it can be constructed from analog devices according to circuit principles, or from digital chips, or in other suitable ways.

[0313] Figure 7 This is a schematic block diagram of an electronic device provided for at least one embodiment of the present disclosure.

[0314] For example, such as Figure 7As shown, the electronic device 700 includes at least one processor 701 and at least one memory 702. The at least one memory 702 includes one or more computer program modules. These computer program modules are stored in the memory 702 and configured to be executed by the at least one processor 701. The one or more computer program modules include instructions for performing the model training method described above. When executed by the at least one processor 701, they can perform one or more steps of the model training method provided in at least one embodiment of this disclosure. The memory 702 and the processor 701 can be interconnected via a bus system and / or other forms of connection mechanisms (not shown).

[0315] For example, processor 701 can be a central processing unit (CPU), digital signal processor (DSP), graphics processing unit (GPU), general-purpose graphics processing unit (GPGPU), artificial intelligence (AI) accelerator, or other form of processing unit with data processing and / or program execution capabilities, such as a field-programmable gate array (FPGA); for example, the central processing unit (CPU) can be an x86, ARM, or RISC-V architecture. Processor 701 can be a general-purpose processor or a special-purpose processor, capable of controlling other components in electronic device 700 to perform desired functions.

[0316] For example, memory 702 may include any combination of one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), USB memory, flash memory, etc.

[0317] Figure 8 This is a schematic block diagram of another electronic device provided for at least one embodiment of the present disclosure.

[0318] The electronic devices in at least one embodiment of this disclosure may include, but are not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable electronic devices, and fixed terminals such as digital TVs and desktop computers. Figure 8 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0319] The electronic device includes at least one processor and a memory. The processor may be referred to as processing device 801 as described below, and the memory may include at least one of ROM 802, RAM 803, and storage device 808 as described below. The memory is used to store programs for performing the methods described in the various method embodiments above; the processor is configured to execute the programs stored in the memory. The processor may include a central processing unit (CPU) or other forms of processing unit having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.

[0320] like Figure 8 As shown, the electronic device 800 may include a processing unit 801 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in ROM 802 or a program loaded from storage device 808 into RAM 803. RAM 803 also stores various programs and data required for the operation of the electronic device 800. The processing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interfaces are also connected to bus 804.

[0321] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, displays, speakers, vibrators, etc.; storage devices 808 including, for example, magnetic tapes, hard disks, etc.; and communication devices 809. Communication device 809 allows electronic device 800 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 An electronic device 800 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0322] In particular, according to at least one embodiment of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, at least one embodiment of this disclosure includes a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such an embodiment, the computer program can be downloaded and installed from a network via a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by a processing device 801, it performs the functions defined in the methods of at least one embodiment of this disclosure.

[0323] It should be noted that the computer-readable medium described above in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In at least one embodiment of this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In at least one embodiment of this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, radio frequency (RF), etc., or any suitable combination thereof.

[0324] The aforementioned computer-readable medium may be included in the aforementioned electronic device 800; or it may exist independently and not assembled into the electronic device 800.

[0325] Figure 9 This is a schematic block diagram of a non-transitory computer-readable storage medium provided for at least one embodiment of the present disclosure.

[0326] For example, such as Figure 9 As shown, a non-transitory computer-readable storage medium 900 stores computer-readable instructions 901, which, when executed by at least one processor, perform one or more steps of the model training method described above.

[0327] For example, the storage medium may include a memory card for a smartphone, a storage component for a tablet computer, a hard drive for a personal computer, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), portable compact disc read-only memory (CD-ROM), flash memory, or any combination of the above storage media, or other suitable storage media. For example, the readable storage medium may also be... Figure 7 The memory 702 in the memory is described in the foregoing content and will not be repeated here.

[0328] Although the present disclosure has been described in detail above with general descriptions and specific embodiments, modifications or improvements can be made to the embodiments of the present disclosure, which will be obvious to those skilled in the art. Therefore, all such modifications or improvements made without departing from the spirit of the present disclosure are within the scope of protection claimed by the present disclosure.

[0329] The following points should be noted regarding this disclosure:

[0330] (1) The accompanying drawings of the embodiments of this disclosure only involve the structures involved in the embodiments of this disclosure. Other structures can be referred to the general design.

[0331] (2) For clarity, the thickness of layers or regions in the drawings used to describe embodiments of the present disclosure is enlarged or reduced, i.e., these drawings are not drawn to actual scale.

[0332] (3) Where there is no conflict, the embodiments of this disclosure and the features in the embodiments can be combined with each other to obtain new embodiments.

[0333] The above description is merely a specific embodiment of this disclosure, but the scope of protection of this disclosure is not limited thereto. The scope of protection of this disclosure should be determined by the scope of protection of the claims.

Claims

1. A model training method, characterized in that, The model training method includes: In response to obtaining the insertion layer configuration corresponding to the first model, the insertion layer information is determined according to the insertion layer configuration, wherein the insertion layer configuration includes parallel strategy information, and the insertion layer information includes the number of insertion layers, the type of insertion layer, and the position information of the insertion layer; The first model is updated based on the insertion layer information to obtain the second model; and The second model is trained in an interleaved pipeline parallel manner. The parallel strategy information includes pipeline parallelism, the number of virtual stages, and the total number of layers in the first model. The step of determining the insertion layer information according to the insertion layer configuration includes: determining the number of insertion layers according to the pipeline parallelism, the number of virtual stages, and the total number of layers in the first model, such that after inserting network layers into the first model, the number of layers handled by a single computing device is an integer multiple of the number of virtual stages.

2. The model training method according to claim 1, characterized in that, The step of determining the insertion layer information according to the insertion layer configuration further includes: In response to the fact that the number of insertion layers is not equal to 0, the insertion layer position information is determined based on the total number of layers in the first model and the number of insertion layers.

3. The model training method according to claim 1, characterized in that, Determining the number of insertion layers based on the pipeline parallelism, the number of virtual stages, and the total number of layers in the first model includes: The number of insertion layers is determined based on the product of the pipeline parallelism and the number of virtual stages, and the total number of layers in the first model.

4. The model training method according to claim 2, characterized in that, Determining the insertion layer position information based on the total number of layers in the first model and the number of insertion layers includes: The insertion interval value is determined based on the total number of layers in the first model and the number of insertion layers; A first number of insertion position indices are determined based on the insertion interval value to obtain the insertion layer position information, wherein the first number is equal to the number of insertion layers.

5. The model training method according to claim 1, characterized in that, The insertion layer configuration further includes at least one of the following: insertion layer enable configuration, insertion layer type configuration, insertion layer position configuration, or custom insertion layer configuration, wherein the insertion layer enable configuration is used to determine whether to enable the insertion strategy. The step of determining the insertion layer information based on the insertion layer configuration in response to obtaining the insertion layer configuration corresponding to the first model further includes: In response to the insertion layer configuration including the insertion layer type configuration, the insertion layer type is determined according to the insertion layer type configuration; In response to the insertion layer configuration including the insertion layer position configuration, and the insertion layer position configuration conforming to a first preset rule, the insertion layer position information is determined based on the insertion layer position configuration. The model training method further includes, after determining the insertion layer information based on the insertion layer configuration: In response to the insertion layer configuration including the custom insertion layer configuration and the custom insertion layer configuration conforming to the second preset rule, the insertion layer information is updated based on the custom insertion layer configuration.

6. The model training method according to claim 1, characterized in that, The insertion layer type includes at least one of the following: an identity placeholder layer, a copy layer, a time-controlling layer, or a memory-controlling layer. The identity placeholder layer does not transform the input data or introduce new parameters during forward propagation; the copy layer is used to perform copy operations on the input data; the control time consumption layer is used to perform cyclic calculations of operators; and the control memory layer is used to simulate actual memory usage by temporarily allocating a buffer.

7. The model training method according to claim 1, characterized in that, The insertion layer position information includes the insertion position index and the relative position marker.

8. The model training method according to claim 1, characterized in that, The step of updating the first model based on the insertion layer information includes: Generate the corresponding insertion layer based on the insertion layer information; The insertion layer is inserted into a specified position in the first model, and the specified position is determined based on the insertion layer position information.

9. The model training method according to claim 1, characterized in that, The step of performing interleaved pipelined parallel training on the second model includes: The second model is divided into multiple layers based on the number of virtual stages and the pipeline parallelism, and the multiple layers are mapped to corresponding computing devices for interleaved pipeline parallel training.

10. The model training method according to claim 1, characterized in that, After performing interleaved pipelined parallel training on the second model, the model training method further includes: Based on the insertion layer information, an insertion layer deletion operation is performed on the trained second model to obtain the first model carrying weights; Save the weights of the first model.

11. The model training method according to claim 1, characterized in that, Before performing interleaved pipelined parallel training on the second model, the model training method further includes: Based on the insertion layer information, perform an insertion layer deletion operation on the second model to obtain the first model; Load the pre-stored weights that match the structure of the first model into the first model; The first model carrying weights is updated based on the insertion layer information to obtain the second model carrying weights.

12. A model training device, characterized in that, The model training device includes: The determination module is configured to determine insertion layer information based on the insertion layer configuration obtained for the first model, wherein the insertion layer configuration includes parallel strategy information, and the insertion layer information includes the number of insertion layers, the type of insertion layer, and the position information of the insertion layer. The update module is configured to update the first model based on the insertion layer information to obtain a second model; and The training module is configured to perform interleaved pipelined parallel training on the second model. The parallel strategy information includes pipeline parallelism, the number of virtual stages, and the total number of layers in the first model. The determining module is further configured to: determine the number of inserted layers based on the pipeline parallelism, the number of virtual stages, and the total number of layers in the first model, such that after inserting network layers into the first model, the number of layers handled by a single computing device is an integer multiple of the number of virtual stages.

13. An electronic device, characterized in that, The electronic device includes: At least one processor; At least one memory, including one or more computer program modules; The one or more computer program modules are stored in the at least one memory and configured to be executed by the at least one processor, and the one or more computer program modules are used to implement the model training method according to any one of claims 1-11.

14. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium stores computer-readable instructions, wherein the computer-readable instructions, when executed by at least one processor, perform the model training method according to any one of claims 1-11.

Images