A multi-modal model pipeline parallel training method for heterogeneous end-side devices

Abstract

This invention discloses a pipelined parallel training method for multimodal models on heterogeneous edge devices. The method includes: constructing a pipelined parallel training architecture for multimodal models on heterogeneous edge devices and addressing the optimization problem of training multimodal models on heterogeneous edge devices; performing modality grouping and device grouping based on an improved genetic algorithm to obtain the optimal modality and device grouping scheme; performing stage division based on a dynamic programming algorithm and calculating the training time of modality grouping as a chromosome fitness evaluation in the genetic algorithm; allocating fusion stages according to the stage division scheme of each modality and adaptively selecting training micro-batches, ultimately obtaining a parallel training scheme for multimodal models on heterogeneous edge devices with the minimum overall training time. This method can fully utilize the parallel branching structure of multimodal models to achieve efficient training of multimodal models on heterogeneous edge devices and optimize the overall training time.

Description

Technical Field

[0001] This invention belongs to the field of parallel training and computation technology for multimodal models, and in particular, it is a pipelined parallel training method for multimodal models for heterogeneous end-devices. Background Technology

[0002] With the rapid development of artificial intelligence technology, traditional unimodal models often process data from a single source in isolation. Multimodal models, by aligning the feature interactions of different modalities, eliminate the ambiguity of unimodal models and enhance robustness. Currently, training of multimodal models is mostly concentrated in the cloud. However, massive amounts of data from different modalities are exploding at the network edge, and uploading all this data to cloud data centers faces challenges related to uplink bandwidth bottlenecks and privacy. Therefore, to overcome bandwidth limitations and protect data privacy, residing data processing tasks on the edge has become an inevitable choice. However, edge devices often lack sufficient memory to support the training of a complete model. Pipeline parallelism, by dividing the model across multiple devices, has become a key technology for solving the memory limitations of model training. However, edge devices are often heterogeneous, and their communication bandwidth is far lower than that of the cloud. Existing pipeline parallel training is mostly designed for model training on homogeneous cloud devices and in high-speed bandwidth environments, and cannot achieve efficient model training in heterogeneous edge scenarios. Furthermore, linearly unfolding the multimodal model leads to excessively long pipeline stages, and increased pipeline bubbles negatively impact training efficiency. Therefore, how to achieve efficient multimodal model training on heterogeneous edge devices has become a key research focus in multimodal model training.

[0003] In the field of pipelined parallel training of models, existing research is constantly optimizing training methods to achieve model training optimization for different models in different scenarios and improve training efficiency. For example, Reference 1 (Jeon B, Wu M, Cao S, et al. Graphpipe: Improving performance and scalability of dnn training with graph pipeline parallelism[C] / / Proceedings of the 30th ACM International Conference on Architectural Support for Programming Languages ​​and Operating Systems, Volume 1. 2025: 557-571.) proposes a method to optimize multimodal model training by utilizing the topology of multimodal models. The aim is to train each modality in parallel using a separate pipeline, thereby achieving modal parallelism and improving the training efficiency of multimodal models. Reference 2 (Ye S, Zeng L, Chu X, et al. Asteroid: Resource-efficient hybrid pipeline parallelism for collaborative dnn training on heterogeneous edge devices[C] / / Proceedings of the 30th Annual International Conference on Mobile Computing and Networking. 2024: 312-326.) proposes a heterogeneous resource-aware model pipeline training method to ensure the effectiveness of parallel pipeline training of models in heterogeneous device scenarios and achieve efficient model training under the constraint of heterogeneous device memory.

[0004] Existing research on multimodal model training varies depending on the application scenario, focusing primarily on optimizing training throughput under homogeneous cloud environments with high communication bandwidth, and attempting to optimize training efficiency using inter-modal parallel structures. However, there is a lack of research that considers both aspects holistically, specifically, research on multimodal parallel training on heterogeneous edge devices. For example, some studies pipeline parallel training for each modality in a multimodal environment without considering the heterogeneity of devices, limited memory, and low communication bandwidth in edge scenarios. Furthermore, simply separating modalities can lead to underutilization of device computing resources. Some studies focus on training models in edge scenarios but fail to consider the multimodal model training scenario, linearly expanding the multimodal models and training each modality sequentially, resulting in longer pipeline training stages, increased pipeline bubbles, and increased device memory pressure. Therefore, how to achieve efficient multimodal model parallel training on edge devices with heterogeneous resources, limited memory, and low communication bandwidth has become a major challenge in multimodal model training. Summary of the Invention

[0005] The purpose of this invention is to provide a pipelined parallel training method for multimodal models on heterogeneous end-side devices, in order to solve the problems in the prior art such as heterogeneous end-side devices, limited memory resources, low end-side communication bandwidth, and insufficient consideration of the topology of multimodal models, and to achieve the goal of optimizing the training throughput of multimodal models on heterogeneous end-side devices while meeting memory constraints.

[0006] The technical solution to achieve the purpose of this invention is: a multimodal model pipeline parallel training method for heterogeneous edge devices, the method comprising the following steps:

[0007] Step 1: Construct a multimodal model pipeline parallel training system architecture for heterogeneous edge devices, and with device memory as a constraint and minimizing the overall training time as the objective function, establish an optimization problem for collaborative training in a heterogeneous environment while meeting model memory requirements;

[0008] Step 2: Measure the performance of the multimodal model on the heterogeneous edge device, extract the operating characteristics of each module of the multimodal model under different configurations, and the state parameters of the heterogeneous edge device; the state parameters include the available memory and inter-device communication bandwidth of the heterogeneous edge device.

[0009] Step 3: Perform the optimal stage partitioning scheme search for the multimodal model in a hybrid parallel manner. Use the improved genetic algorithm to perform collaborative grouping of modalities and devices, and combine dynamic programming to determine the stage partitioning within each modal group to obtain the optimal grouping and stage partitioning scheme.

[0010] Step 4: Allocate equipment for the fusion stage according to the optimal grouping and stage division scheme, calculate the overall training time of the optimal scheme, adaptively obtain the optimal training micro-batch, and output a comprehensive parallel training scheme that includes modality grouping, stage division within each modality group, and micro-batch allocation within each stage.

[0011] Furthermore, the memory requirements in step 1 are quantified using a memory model, including:

[0012] Static memory requirements: Parameters that continuously occupy device memory and whose size remains basically unchanged during training;

[0013] Dynamic memory requirements: Memory for intermediate activation values ​​generated during the forward propagation process and related to micro-batches and network dimensions;

[0014] Instantaneous memory requirements: Workspace memory temporarily generated during backpropagation and occupied before being released;

[0015] Stage memory requirements: determined by the static memory requirements, dynamic memory requirements, instantaneous memory requirements, and the number of micro-batches in transit in the pipeline; wherein, micro-batches in transit refer to micro-batches that have completed forward propagation but have not completed backward propagation calculations.

[0016] Furthermore, the total training time in step 1 is quantified using a training time model, including:

[0017] Training readiness time: determined based on the arrival time of intermediate results from the previous stage;

[0018] Training start time: determined based on the training readiness time and equipment idle time.

[0019] Transmission time: determined based on the amount of data transmitted and the transmission rate; when two stages are trained on the same device, the intermediate result transmission time is ignored, i.e., the transmission time is 0.

[0020] Transmission readiness time: determined based on the completion time of each training phase;

[0021] Transmission start time: determined based on transmission readiness time and transmission resource idle time;

[0022] Overall training time: determined based on the completion time of the last micro-batch training in each stage.

[0023] Furthermore, the multimodal model pipelined parallel training system architecture for heterogeneous edge devices described in step 1 includes three methods: modal parallelism, pipelined parallelism, and data parallelism; among which,

[0024] Modal parallelism is adopted between modalities: the modalities of the multimodal model can be independently parallelized. During the training of the multimodal model, different modalities are assigned to different devices for parallel training. During the fusion stage, the outputs of each modality are aligned through a contrastive learning mechanism.

[0025] The modality adopts a pipelined parallel approach: the layers in a single modality group are divided into multiple stages and distributed to different devices. The different stages of the modality are trained sequentially in a pipeline manner. After the device completes the training of a micro-batch, the intermediate results are transmitted to the next stage while the next micro-batch of the current stage is trained in parallel.

[0026] The data parallelism approach is adopted within the phase: within the modal phase, the input micro-batch is split and distributed to multiple devices for training of their respective batches, and gradient synchronization is performed after a complete batch is trained.

[0027] Furthermore, the optimization problem of collaboratively training the multimodal model by heterogeneous end-side devices in step 1 can be expressed as:

[0028]

[0029] constraint Due to memory constraints, it is necessary to ensure that the memory requirements allocated by the device during a given period do not exceed the available memory of the device; constraint The phase division constraint ensures that each module of the modality is assigned to a specific phase, and that there is no overlap between phases; Assign constraints to devices, requiring at least one device to be allocated for each stage to support stage training; constraints Each stage is required to be responsible for training only one stage, in order to avoid stages being trained on the same device, which would affect the efficiency of pipelined parallel training.

[0030] In the formula, Minimize the total training time , For equipment The allocated stage memory requirements, For equipment Available memory, Represents the k-th mode The Middle Phase Corresponding distribution device set The Middle One device ; Indicates the total number of stages. Represent each mode An ordered set of modules Representing the k-th mode respectively The first in The first stage, the first The number of stages; K represents the total number of modes. Indicates equipment Whether to be assigned to a stage Equipment collection .

[0031] Further, step 2 extracts runtime features, specifically including:

[0032] On different devices, each module of the multimodal model is simulated to generate adapted inputs according to the input specifications. The forward and backward time, input and output data volume, parameter scale and activation value memory usage of each module of the target multimodal model are measured and recorded on different devices as the micro-batch b changes.

[0033] Calculate the memory requirements at each level under different micro-batch sizes.

[0034] Furthermore, step 3 specifically includes:

[0035] An improved genetic algorithm is used for modal grouping and heterogeneous device allocation for multimodal applications;

[0036] For the selectable micro-batch size, modal grouping and device grouping are jointly genetically encoded. After standardization to eliminate equivalent redundant solutions, feasibility repair is performed based on device memory constraints.

[0037] The dynamic programming method is used to divide the chromosome individual into stages and calculate the training time, and the chromosome fitness is evaluated. At the same time, a fitness cache is established to reuse the evaluation results of the same standardized individuals.

[0038] Through continuous iterative evolution via selection, crossover, and mutation operations, the optimal grouping scheme for overall training time and its corresponding stage division results are generated for each candidate micro-batch.

[0039] Furthermore, in step 3, joint gene encoding is performed on modality grouping and device grouping. The specific process includes:

[0040] A chromosome with a two-layer hybrid coding structure is constructed, the total number of gene loci being the sum of the number of modalities and the number of devices; the front segment of the chromosome represents the modal grouping scheme, each bit in the front segment corresponds to a modality, and the value represents the modality group number to which the modality is assigned; the back segment of the chromosome represents the device grouping scheme, each bit in the back segment corresponds to a device, and the value represents the modality group number to which the device is assigned.

[0041] Furthermore, in step 3, standardization eliminates equivalent redundant solutions. The specific process includes:

[0042] The modal grouping codes of the first segment of the chromosome are relabeled according to the order of first appearance to obtain a canonical representation. Based on this, the device grouping codes of the second segment are uniformly mapped and invalid pointers are corrected. After integrity verification, device reassignment is performed on the modal groups of unassigned devices to meet the constraints, thereby merging multiple equivalent codes into a unique canonical code to reduce search space redundancy.

[0043] Furthermore, in step 3, feasibility repairs are performed based on device memory constraints, specifically including:

[0044] Based on the device grouping results, calculate the memory supply and demand of each modal group, and mark the modal group with insufficient supply (i.e., supply less than the preset threshold) as the memory-insufficient group, and the other group as the memory-sufficient group.

[0045] Migrate devices from modal groups with ample memory and more than one device to groups with insufficient memory and update supply and demand values. Repeat the current process until all modal groups meet memory constraints or reach the preset repair limit. If there are still modal groups that have not been allocated devices, perform fallback allocation while keeping the remaining modal groups feasible to ensure that each effective modal group corresponds to at least one device and obtain feasible chromosome codes that meet memory constraints.

[0046] Furthermore, in step 3, the criteria for dividing the stages in dynamic programming are as follows:

[0047] For each modality group, a dynamic programming recursion is constructed with the goal of minimizing the pipeline bottleneck time. During the recursion process, if multiple candidate splitting schemes have the same bottleneck time, the scheme with the smaller stage delay is further selected. After each modality group has completed stage division, the total training time is calculated, and this total training time is used as the fitness value of the corresponding chromosome individual. The bottleneck time is the larger value between the intra-stage computation time and the inter-stage communication time, and the stage delay is the sum of the computation time and the bottleneck time.

[0048] Furthermore, in step 3, the dynamic programming stage division includes data parallelism and micro-batch adjustment mechanisms:

[0049] Within each candidate stage, continuous module intervals are allowed to be deployed to multiple devices to form data parallel subgroups. The total amount of stage micro-batch is non-uniformly divided into sub-micro-batch according to the difference in device computing power, and memory feasibility is verified.

[0050] Furthermore, in step 3, the dynamic programming phase division also includes a communication compression mechanism:

[0051] When the stage computation time is less than the stage communication time, the compression algorithm library is traversed to select the compression method that minimizes the stage bottleneck time in order to balance computation and communication overhead.

[0052] Furthermore, in step 4, the equipment allocation for the fusion phase includes the following specific processes:

[0053] After the modality phase is divided, the device containing the last phase of the modality that satisfies memory constraints and has the longest training time for a single micro-batch is selected, and the multimodal fusion phase is deployed.

[0054] Compared with the prior art, the significant advantages of this invention are:

[0055] (1) This invention breaks through the limitation of the traditional “linear expansion” of multimodal models and uses the natural parallel structure of multimodal models to group modalities. This approach effectively shortens the pipeline depth, reduces the number of micro-batches in transit, thereby significantly alleviating the memory pressure on edge devices, reducing pipeline bubble overhead, and improving the overall training throughput.

[0056] (2) To address the pain point of significant differences between the computing power and memory capacity of edge devices, this invention uses a dynamic programming algorithm to jointly optimize stage partitioning, data parallel configuration, and "non-uniform" micro-batch allocation. This minimizes the bottleneck time of each stage and achieves efficient collaboration of heterogeneous resources.

[0057] (3) In a low-bandwidth edge environment, this invention actively avoids the boundaries of modules with large data volumes when dividing the stages and introduces a "communication compression mechanism". By moderately increasing the computational overhead to reduce the amount of communication data, the invention effectively solves the problem of low bandwidth restricting training efficiency.

[0058] (4) This invention defines a refined memory model (covering static, dynamic, and instantaneous memory) and a training time model. By performing performance measurement and feature extraction on actual devices, it provides accurate data support for the search of parallel solutions, ensuring the feasibility and optimality of the solutions.

[0059] (5) Improved genetic algorithms are used for collaborative grouping of modalities and devices, and "standardization to eliminate equivalent redundant solutions" and "feasibility repair" mechanisms are introduced. These techniques significantly reduce the search space and avoid invalid evaluations, thereby enabling the optimal training scheme to be locked more quickly.

[0060] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0061] Figure 1 This is a flowchart of a multimodal model pipeline parallel training method for heterogeneous end-side devices in one embodiment.

[0062] Figure 2 This is a comparison chart of the training effects of a multimodal model pipeline parallel training method for heterogeneous edge devices in one embodiment and four mainstream methods in a heterogeneous edge environment. Detailed Implementation

[0063] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0064] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0065] In one embodiment, a pipelined parallel training method for multimodal models on heterogeneous edge devices is provided. This method performs inter-modal modal union, intra-modal pipelined parallelism, and intra-stage data parallelism on a multimodal model, minimizing the overall training time of the multimodal model while considering the heterogeneity of edge device resources and the limitations of memory and communication bandwidth. The method uses an improved genetic algorithm to group modalities and devices, then performs stage partitioning based on dynamic programming. The overall training time is calculated based on the partitioning scheme as a chromosome fitness evaluation. Through iterative evolution via selection, crossover, and mutation operations, the optimal grouping scheme and its corresponding stage partitioning results are generated for each candidate micro-batch, achieving the goal of optimizing the overall training time of the multimodal model on the edge device.

[0066] Combination Figure 1 The method includes the following steps:

[0067] Step 1: Construct a multimodal model pipeline parallel training system architecture for heterogeneous edge devices, and with device memory as a constraint and minimizing the overall training time as the objective function, establish an optimization problem for collaborative training in a heterogeneous environment while meeting model memory requirements;

[0068] Step 2: Measure the performance of the multimodal model on the heterogeneous edge device, extract the operating characteristics of each module of the multimodal model under different configurations, and the state parameters of the heterogeneous edge device; the state parameters include the available memory and inter-device communication bandwidth of the heterogeneous edge device.

[0069] Step 3: Perform the optimal stage partitioning scheme search for the multimodal model in a hybrid parallel manner. Use the improved genetic algorithm to perform collaborative grouping of modalities and devices, and combine dynamic programming to determine the stage partitioning within each modal group to obtain the optimal grouping and stage partitioning scheme.

[0070] Step 4: Allocate equipment for the fusion stage according to the optimal grouping and stage division scheme, calculate the overall training time of the optimal scheme, adaptively obtain the optimal training micro-batch, and output a comprehensive parallel training scheme that includes modality grouping, stage division within each modality group, and micro-batch allocation within each stage.

[0071] Furthermore, in one embodiment, the multimodal model pipelined parallel training system architecture for heterogeneous edge devices described in step 1 includes three methods: modal parallelism, pipelined parallelism, and data parallelism; wherein,

[0072] Modal parallelism is adopted between modalities: the modalities of the multimodal model can be independently parallelized. During the training of the multimodal model, different modalities are assigned to different devices for parallel training. During the fusion stage, the outputs of each modality are aligned through a contrastive learning mechanism.

[0073] The modality adopts a pipelined parallel approach: the layers in a single modality group are divided into multiple stages and distributed to different devices. The different stages of the modality are trained sequentially in a pipeline manner. After the device completes the training of a micro-batch, the intermediate results are transmitted to the next stage while the next micro-batch of the current stage is trained in parallel.

[0074] The data parallelism approach is adopted within the phase: within the modal phase, the input micro-batch is split and distributed to multiple devices for training of their respective batches, and gradient synchronization is performed after a complete batch is trained.

[0075] Furthermore, in one embodiment, step 1 further includes defining and establishing a system model, a training phase model, a training time model, and a memory model.

[0076] Preferably, in some embodiments, a system model is defined, including an edge device model and a multimodal neural network model, specifically including:

[0077] The system consists of Heterogeneous end-side devices Integration, the q-th heterogeneous end-side device is the heterogeneous device. Different memory resources and computing power To jointly train a system with It consists of one modal branch and one fusion part, represented as ,in Indicates the fusion part, Indicates the first Each mode. Depend on It consists of ordered modules, represented as For modes Micro-batch size is One of the modules It has static memory Dynamic memory Instantaneous memory In the equipment Forward propagation time In the equipment Backpropagation time Input the size of intermediate results and output intermediate size Feature parameters.

[0078] Preferably, in some embodiments, establishing a training phase model specifically includes:

[0079] When performing model training in a pipelined parallel manner, continuous modalities need to be divided into stages and assigned to different devices to achieve pipelined parallel training. Each modality Classified as A sequence of stages, represented as

[0080]

[0081] Each stage (modal) The z-th stage corresponds to a set of devices. , Representation phase The first in the set of allocated devices Each device. The output size of a stage is the output size of the last layer within that stage, and the input size is the input size of the first layer within that stage. Each stage... It must meet the following conditions:

[0082]

[0083] That is, each module belongs to only one stage, and the stage covers all layers. Each stage is assigned a set of devices, and the set of devices trains the stage in a data-parallel manner.

[0084] Define 0-1 variables This represents the mapping from modules to stages:

[0085]

[0086] in, Representing modes Modules in Divided into stages Conversely, it indicates that it does not belong to a stage. This belongs to another stage.

[0087] In addition, define 0-1 variables This indicates the mapping from device to stage:

[0088]

[0089] in, Indicates equipment Assigned to stage equipment group Conversely, it means that it has not been allocated to the current stage.

[0090] Preferably, in some embodiments, the memory requirement in step 1 is quantified using a memory model to ensure that the available memory on the device can meet the allocated training stage requirements. The memory requirement includes:

[0091] Static memory requirements: Parameters that continuously occupy device memory and whose size remains basically unchanged during training;

[0092] Dynamic memory requirements: Memory for intermediate activation values ​​generated during the forward propagation process and related to micro-batches and network dimensions;

[0093] Instantaneous memory requirements: Workspace memory temporarily generated during backpropagation and occupied before being released;

[0094] Stage memory requirements: determined by the static memory requirements, dynamic memory requirements, instantaneous memory requirements, and the number of micro-batches in transit in the pipeline; wherein, micro-batches in transit refer to micro-batches that have completed forward propagation but have not completed backward propagation calculations.

[0095] Specifically, static memory These are parameters that continuously occupy device memory during training and whose size remains basically unchanged. They include network layer weight parameters, bias parameters, and state variables such as gradient momentum and variance maintained by the optimizer, and are independent of the micro-batch size.

[0096] Dynamic memory It is the memory space occupied by intermediate activation values ​​generated and maintained during the forward propagation process in training, and its size is similar to that of the micro-batch. And related to the output dimension of the network layer.

[0097] instantaneous memory This refers to the memory occupied by the gradient tensors temporarily generated during backpropagation, the inter-device communication buffer, and the workspace required for gradient aggregation operations. This memory is released after gradient synchronization or parameter updates, and its size is similar to that of the micro-batch. Linear correlation.

[0098] The system training architecture distributes the multimodal model into several modality groups, enabling parallel modal training between modality groups. Within a modality, the modality group module is divided into different stages for pipelined parallel training, processing a training batch... Divided into The size is Micro-batch, i.e.:

[0099]

[0100] The memory model includes:

[0101] When allocating stages, it is necessary to ensure that the device memory is sufficient to support the training of the allocated stages. Stage memory consists of static memory, dynamic memory, and transient memory, represented as follows:

[0102]

[0103] In the formula, Representing modes The z-th stage corresponds to the stage memory, static memory, dynamic memory, and transient memory;

[0104] The static memory is the cumulative value of the static memory of each module within a stage, expressed as:

[0105]

[0106] In the formula, Representing modes Mid-stage The nth module Static memory;

[0107] Dynamic memory is the product of the dynamic memory of each module in the next stage of the corresponding micro-batch and the number of micro-batches in transit, expressed as:

[0108]

[0109] In the formula, For the stage The number of micro-batch shipments in transit. Representing modes Mid-stage The nth module Dynamic memory; In-transit micro-batches refer to micro-batches that have completed forward propagation but not backward propagation calculations, and their value is the number of remaining stages in the pipeline.

[0110] Instantaneous memory is the maximum value of the product of the instantaneous memory of each module within a stage and the number of micro-batches in transit, expressed as:

[0111]

[0112] In the formula, Representing modes Mid-stage The nth module Instantaneous memory.

[0113] Because of data parallelism within a stage, the stage memory pressure is distributed to the data parallel devices. Since static memory contains the model's fundamental parameters, the data parallel devices store the complete static parameters, while dynamic memory and transient memory are related to the allocated batch size, i.e.:

[0114]

[0115] in, , respectively equipment The allocated stage memory requirements and batch size, among which To ensure the device can support phased training, the allocated memory requirements must be guaranteed. Less than or equal to the device's available memory ,Right now:

[0116] .

[0117] Preferably, in some embodiments, the total training time in step 1 is quantified using a training time model, including:

[0118] Training readiness time: determined based on the arrival time of intermediate results from the previous stage;

[0119] Training start time: determined based on the training readiness time and equipment idle time.

[0120] Transmission time: determined based on the amount of data transmitted and the transmission rate; when two stages are trained on the same device, the intermediate result transmission time is ignored, i.e., the transmission time is 0.

[0121] Transmission readiness time: determined based on the completion time of each training phase;

[0122] Transmission start time: determined based on transmission readiness time and transmission resource idle time;

[0123] Overall training time: determined based on the completion time of the last micro-batch training in each stage.

[0124] Specifically, the training time model includes:

[0125] (1) The phase training process includes forward training time. Reverse training time Forward transmission time and reverse transmission time .

[0126] The forward and backward training times are the sum of the forward and backward training times for each module in the stage, and are expressed as follows:

[0127] ,

[0128] In the formula, Representing modes Mid-stage The nth module Forward training time, Representing modes Mid-stage The nth module Backward training time;

[0129] Forward and reverse transmission times depend on the transmission rate and the size of intermediate results, respectively, as follows:

[0130] ,

[0131] In the formula, and They are stages The amount of intermediate result data output by the last module and the amount of intermediate result data input by the first module, Indicates equipment and The communication rate between them.

[0132] Because of the pipelined parallel stage, during stage training, the equipment may be transmitting data and training simultaneously, thus creating a bottleneck time for each stage. for:

[0133]

[0134] Stage delay The time required to process a micro-batch in a stage is expressed as:

[0135]

[0136] (2) The overall training time depends on the start and end times of each phase. and The training time and transmission time are respectively represented as follows: The training readiness time for each stage is represented as follows:

[0137] ,

[0138] In the formula, , They represent the first In each micro-batch The first in modality The ready time for each stage of forward and backward training , They represent the first time. In each micro-batch The completion time of the forward transmission in the previous stage (z-1) and the completion time of the transmission in the next stage (z+1) are the current stages. The time points at which input data is received during forward and backward training; Indicates equipment Free time.

[0139] The completion times for the forward and backward training phases are respectively expressed as follows: , :

[0140] ,

[0141] Similarly, the ready times for the forward and reverse transmission processes of the device are expressed as follows: , :

[0142] ,

[0143] in Indicates the idle time point of the device transmission;

[0144] The phase forward and backward transmission completion times are respectively expressed as: , :

[0145] ,

[0146] The above process does not consider the fusion phase. The fusion mode, as the final stage of merging various modes, involves fewer network layers and requires less computation and memory. It waits for the intermediate results from all modes to arrive before performing forward and backward propagation. The model training method uses a 1F1B approach, meaning the device alternately executes a micro-batch of backward propagation after completing one micro-batch of forward propagation. Therefore, the last mode to complete training is the backward propagation process of the first stage, i.e., the bus training completion time is [time missing]. The mode phase with the longest completion time in each micro-batch is represented as:

[0147]

[0148] In the formula, Indicates the first In each micro-batch The time to complete the reverse training of the first stage in the previous modality;

[0149] Training throughput (TPS) is the number of training samples per unit of time, expressed as:

[0150]

[0151] Preferably, in some embodiments, the optimization problem of collaboratively training a multimodal model by heterogeneous end-side devices in step 1 is expressed as:

[0152]

[0153] constraint Due to memory constraints, it is necessary to ensure that the memory requirements allocated by the device during a given period do not exceed the available memory of the device; constraint The phase division constraint ensures that each module of the modality is assigned to a specific phase, and that there is no overlap between phases; Assign constraints to devices, requiring at least one device to be allocated for each stage to support stage training; constraints Each stage is required to be responsible for training only one stage, in order to avoid stages being trained on the same device, which would affect the efficiency of pipelined parallel training.

[0154] In the formula, Minimize the total training time , For equipment The allocated stage memory requirements, For equipment Available memory, Represents the k-th mode The Middle Phase Corresponding distribution device set The Middle One device ; Indicates the total number of stages. Represent each mode An ordered set of modules Representing the k-th mode respectively The first in The first stage, the first The number of stages; K represents the total number of modes. Indicates equipment Whether to be assigned to a stage Equipment collection .

[0155] Furthermore, in one embodiment, step 2 extracts operational features, specifically including:

[0156] On different devices, each module of the multimodal model is simulated to generate adapted inputs according to the input specifications. The forward and backward time, input and output data volume, parameter scale and activation value memory usage of each module of the target multimodal model are measured and recorded on different devices as the micro-batch b changes.

[0157] Calculate the memory requirements at each level under different micro-batch sizes.

[0158] Preferably, in some embodiments, the process of extracting model operation features and heterogeneous end-side device state parameters in step 2 is as follows:

[0159] Step 2.1: For each device in the heterogeneous end-side device set, a benchmark model module is run on the device and the execution time is recorded. The relative computing power of each device is determined by the time ratio relative to the benchmark device, providing a basis for non-uniform micro-batch allocation based on the differences in device computing power in subsequent stages. Simultaneously, a bidirectional transmission test with a fixed amount of data is performed between any two devices. The actual communication bandwidth between the devices is calculated based on the transmission time and data volume, thereby obtaining the communication rate parameters between the heterogeneous end-side devices.

[0160] Step 2.2 involves extracting runtime features from the target multimodal model. On heterogeneous devices of different types, the computational modules within each modal branch are independently profiled. During profiling, a complete forward and backward propagation of the model is first performed by injecting a small-scale exploratory input to capture the shape, specifications, and data type of the input tensors received by each computational layer. Based on these input specifications, a virtual input matching the target batch size is constructed for each candidate micro-batch size. The forward and backward propagation of this module are simulated, and the forward and backward propagation times, as well as the data size of the module's output tensor, are recorded.

[0161] Step 2.3 involves memory feature extraction. After the model is loaded onto the target device and the input specification pre-probe is completed, the memory usage of the model under test is measured independently layer by layer. For each computational layer, its trainable parameters are traversed, and the product of the number of elements in each parameter tensor and the size of a single element in bytes is accumulated to obtain the total number of parameter bytes for that layer, which is used as the static memory usage for that layer. Subsequently, before constructing a virtual input tensor corresponding to the target micro-batch size and performing forward propagation, the current allocated memory of the device is recorded as a baseline value. After the forward propagation is completed, the allocated memory of the device is recorded again, and the difference between the two allocated memory values ​​is determined as the dynamic activation memory usage generated during the forward propagation of that layer. At the same time, the instantaneous peak value of device memory allocation during forward propagation is monitored and recorded, and the difference between the peak memory value and the allocated memory value after propagation is taken as the instantaneous memory usage temporarily allocated and released during the computation of that layer. Through the above-described layered independent measurement process, the model's operational characteristic parameters, such as forward time, backward time, output data volume, static memory usage, dynamic activation memory usage, and instantaneous memory usage, are obtained for each layer at a specified micro-batch size.

[0162] Furthermore, in one embodiment, step 3 specifically includes:

[0163] An improved genetic algorithm is used for modal grouping and heterogeneous device allocation for multimodal applications;

[0164] For the selectable micro-batch size, modal grouping and device grouping are jointly genetically encoded. After standardization to eliminate equivalent redundant solutions, feasibility repair is performed based on device memory constraints.

[0165] The dynamic programming method is used to divide the chromosome individual into stages and calculate the training time, and the chromosome fitness is evaluated. At the same time, a fitness cache is established to reuse the evaluation results of the same standardized individuals.

[0166] Through continuous iterative evolution via selection, crossover, and mutation operations, the optimal grouping scheme for overall training time and its corresponding stage division results are generated for each candidate micro-batch.

[0167] Preferably, in some embodiments, in step 3, joint gene encoding is performed on modality grouping and device grouping, specifically including:

[0168] A chromosome with a two-layer hybrid coding structure is constructed, the total number of gene loci being the sum of the number of modalities and the number of devices; the front segment of the chromosome represents the modal grouping scheme, each bit in the front segment corresponds to a modality, and the value represents the modality group number to which the modality is assigned; the back segment of the chromosome represents the device grouping scheme, each bit in the back segment corresponds to a device, and the value represents the modality group number to which the device is assigned.

[0169] Preferably, in some embodiments, step 3, standardization to eliminate equivalent redundant solutions, specifically includes the following process:

[0170] The modal grouping codes of the first segment of the chromosome are relabeled according to the order of first appearance to obtain a canonical representation. Based on this, the device grouping codes of the second segment are uniformly mapped and invalid pointers are corrected. After integrity verification, device reassignment is performed on the modal groups of unassigned devices to meet the constraints, thereby merging multiple equivalent codes into a unique canonical code to reduce search space redundancy.

[0171] Preferably, in some embodiments, step 3, performing feasibility repair based on device memory constraints, specifically includes:

[0172] Based on the device grouping results, calculate the memory supply and demand of each modal group, and mark the modal group with insufficient supply (i.e., supply less than the preset threshold) as the memory-insufficient group, and the other group as the memory-sufficient group.

[0173] Migrate devices from modal groups with ample memory and more than one device to groups with insufficient memory and update supply and demand values. Repeat the current process until all modal groups meet memory constraints or reach the preset repair limit. If there are still modal groups that have not been allocated devices, perform fallback allocation while keeping the remaining modal groups feasible to ensure that each effective modal group corresponds to at least one device and obtain feasible chromosome codes that meet memory constraints.

[0174] Preferably, in some embodiments, the criteria for dividing the stages in step 3 of dynamic programming are as follows:

[0175] For each modality group, a dynamic programming recursion is constructed with the goal of minimizing the pipeline bottleneck time. During the recursion process, if multiple candidate splitting schemes have the same bottleneck time, the scheme with the smaller stage delay is further selected. After each modality group has completed stage division, the total training time is calculated, and this total training time is used as the fitness value of the corresponding chromosome individual. The bottleneck time is the larger value between the intra-stage computation time and the inter-stage communication time, and the stage delay is the sum of the computation time and the bottleneck time.

[0176] Preferably, in some embodiments, step 3, the dynamic programming stage division includes data parallelism and micro-batch adjustment mechanisms:

[0177] Within each candidate stage, continuous module intervals are allowed to be deployed to multiple devices to form data parallel subgroups. The total amount of stage micro-batch is non-uniformly divided into sub-micro-batch according to the difference in device computing power, and memory feasibility is verified.

[0178] Specifically, within each candidate stage, continuous module intervals are allowed to be deployed to multiple devices to form data parallel subgroups. The total number of micro-batches in this stage is non-uniformly divided into multiple sub-micro-batches based on the differences in device computing power and allocated to each device within the group. During allocation, the sub-micro-batch sizes are matched according to the device computing power ranking, so that devices with stronger computing power (computing power higher than the preset computing power threshold) undertake more (more than the preset sub-micro-batch number threshold) sub-micro-batches, while devices with weaker computing power (computing power lower than the preset computing power threshold) undertake fewer (less than the preset sub-micro-batch number threshold) sub-micro-batches, in order to balance the computing load of each device within the stage. At the same time, the dynamic memory usage within the stage is estimated by combining the number of micro-batches in transit, and the memory feasibility of the sub-micro-batch allocation scheme is verified. For all allocation combinations that meet the memory constraints, the bottleneck cost and stage latency of the stage are calculated, and the combination that optimizes the performance of the stage is selected as the device batch configuration for dynamic programming state transition, thereby achieving joint optimization of stage division, data parallelism within the stage, and non-uniform allocation of micro-batches.

[0179] Preferably, in some embodiments, step 3, the dynamic programming stage division further includes a communication compression mechanism:

[0180] When the stage computation time is less than the stage communication time, the compression algorithm library is traversed to select the compression method that minimizes the stage bottleneck time in order to balance computation and communication overhead.

[0181] Specifically, when the stage computation time is less than the stage communication time, the communication compression algorithm library is traversed. For each candidate compression method, the stage computation time and communication time are adjusted according to its compression ratio and additional computational overhead. Under the constraint of memory requirements, the communication compression algorithm with the minimum stage bottleneck time and the minimum stage latency is selected.

[0182] Here, a communication compression mechanism is introduced to avoid excessively long communication times from affecting the overall training time. By compressing intermediate results, the computation time and memory requirements are increased, thereby reducing transmission time and achieving a balance between communication time and computation time.

[0183] For example, in some embodiments, step 3 performs modality and device collaborative grouping based on an improved genetic algorithm, specifically including:

[0184] Step 3.1: Construct a chromosome with a two-layer hybrid coding structure. The total number of gene loci is the sum of the number of modalities and the number of devices. The front segment of the chromosome represents the modal grouping scheme. Each bit in the front segment corresponds to a modality, and the value represents the modal group number to which the modality is assigned. The back segment of the chromosome represents the device grouping scheme. Each bit in the back segment represents a device, and the value represents the modal group number to which the device is assigned.

[0185] Step 3.2: After determining the chromosome coding structure, for each gene, generate a random code within the modality range, and then standardize the gene codes to eliminate equivalent redundant solutions. The modality grouping codes of the first segment of the chromosome are relabeled according to the order of their first occurrence to obtain a standardized representation. Based on this, the device grouping codes of the second segment are uniformly mapped and invalid pointers are corrected. Their codes are also mapped to standard groups. Codes assigned to non-existent modality groups are re-randomized, with the generation range being the number of modality groups.

[0186] Step 3.3: Perform feasibility repair on chromosomes based on device memory constraints to compress the search space and avoid invalid evaluations. The total available device memory within each modality group is calculated based on the device grouping results as the memory supply. This is combined with the sum of static, dynamic, and instantaneous memory for each layer within the modality group at the target micro-batch size to obtain the memory requirement. Modality groups with memory supply less than requirement are marked as memory-deficient groups. For each memory-deficient group, devices are selected from other modality groups with surplus memory supply and more than one allocated device and migrated to that group. The supply and demand values ​​for each group after migration are updated synchronously. This judgment and migration operation is repeated until all modality groups meet the memory constraints or the preset maximum number of repair iterations is reached. If any modality group still has no allocated devices when the repair terminates, a fallback device allocation is performed while maintaining the memory feasibility of other modality groups, ensuring that each valid modality group corresponds to at least one device. After the above feasibility repair, feasible chromosome individuals that meet the device memory constraints are finally obtained.

[0187] Step 3.4 involves performing dynamic programming to divide the training into stages and calculate the training time, specifically including:

[0188] Step 3.4.1: For each chromosome individual in the genetic algorithm population, firstly, parse the modality grouping scheme and device grouping scheme determined by its encoding to obtain the module sequence contained in each modality group and the set of heterogeneous devices allocated to that modality group. Within each modality group, with the objective of minimizing pipeline bottleneck time, use dynamic programming to solve for the optimal stage division, parallel configuration of device data within the stage, and joint scheme of the sub-micro-batch size undertaken by each device.

[0189] The state of dynamic programming is defined by two dimensions: the number of allocated devices and the number of modules in the allocated stages. The state value represents the minimum bottleneck time that can be achieved under the corresponding device and module scale. At the same time, it records the module splitting position of the previous stage, the number of devices occupied in the current stage, the sub-micro-batch size allocated to each device, the communication compression strategy adopted, and the latency cost of this stage.

[0190] Step 3.4.2: During the recursive process, for each state, enumerate the number of devices to be used in the current stage and the module splitting position of the previous stage, and assign the continuous module sequence located between the splitting position and the current module to the corresponding device subgroup to form a new candidate pipeline stage.

[0191] Step 3.4.3 involves adjusting the micro-batch allocation for each stage. Based on the modules and devices allocated to each stage, the total number of micro-batches for that stage is non-uniformly divided into multiple sub-micro-batches and distributed to each device within the group according to the differences in device computing power. During allocation, the sub-micro-batch sizes are matched according to the device computing power ranking, so that devices with stronger computing power undertake more sub-micro-batches and devices with weaker computing power undertake fewer sub-micro-batches, thereby balancing the computing load of each device within the stage. At the same time, the dynamic memory usage within the stage is estimated based on the number of micro-batches in transit, and the memory feasibility of the sub-micro-batch allocation scheme is verified. For all allocation combinations that meet the memory constraints, the bottleneck cost and stage latency of the stage are calculated, and the combination that optimizes the performance of the stage is selected as the device batch configuration for dynamic programming state transition, thereby achieving joint optimization of stage division, data parallelism within the stage, and non-uniform micro-batch allocation.

[0192] Step 3.4.4: Select a communication compression strategy for the divided stages. For the current stage, when the stage computation time is less than the stage communication time, traverse the communication compression algorithm library. For each candidate compression method, adjust the stage computation time and communication time according to its compression ratio and additional computational overhead. Under the constraint of meeting memory requirements, select the communication compression algorithm with the minimum stage bottleneck time and the minimum stage latency.

[0193] Step 3.5: Perform chromosome fitness assessment. Based on the optimal stage division of each modality group obtained by dynamic programming, calculate the predicted training time, take the longest training time of the modality group as the chromosome fitness, and cache the corresponding chromosome code and fitness to reuse the assessment results of the same standardized individuals and avoid repeatedly calculating the same chromosome fitness.

[0194] Step 3.6 involves iterative evolution through selection, crossover, and mutation operations on the initially randomly generated chromosome individuals. The specific process includes:

[0195] Step 3.6.1: Select superior chromosomes. Evaluate the fitness of individuals with chromosomes in the initially generated population, arranging them in descending order of fitness. Individuals with higher fitness are retained and directly enter the next generation. For the remaining individuals, a tournament selection strategy is used to randomly select two parent individuals from the current population with higher fitness.

[0196] Step 3.6.2: Crossover to generate new chromosomes. Perform a single-point crossover operation at randomly generated crossover points to exchange parent gene segments to generate offspring chromosomes.

[0197] Step 3.6.3: Perform mutation operation, and mutate random gene loci on the offspring chromosome with a preset mutation probability. The mutation range is the current effective modality group number.

[0198] Step 3.7: Call the feasibility repair function on the offspring chromosomes after crossover and mutation operations to eliminate invalid grouping codes and meet device memory constraints. Add the repaired offspring individuals to the new generation population until the population size is restored to the preset size. Repeat the above evaluation, selection, crossover, mutation, and repair process, and after multiple generations of iteration, output the individual with the best fitness in the population as the final grouping scheme.

[0199] Furthermore, in one embodiment, step 4 involves device allocation during the fusion phase, specifically including:

[0200] After the modality phase is divided, the device containing the last phase of the modality that satisfies memory constraints and has the longest training time for a single micro-batch is selected, and the multimodal fusion phase is deployed.

[0201] Preferably, in some embodiments, the adaptive selection of micro-batches in step 4 is as follows: for multiple candidate micro-batch sizes, a hybrid parallel optimal solution search is performed independently, and the results of each candidate solution are compared. The micro-batch that produces the minimum global training time is selected as the final execution parameter.

[0202] In summary, step 4 is as follows:

[0203] After modal grouping and stage partitioning using an improved genetic algorithm and dynamic programming, a modal grouping and stage partitioning scheme with the minimum overall training time is obtained. The scheme selects the most time-consuming modality on the last stage device to prevent further increases in transmission time for that modality. Based on the aforementioned overall training time... The formulas for calculating the training throughput and the optimal overall training time and throughput for the current micro-batch are used to adaptively select the micro-batch with the shortest training time. Finally, the optimal training micro-batch and a comprehensive parallel training scheme including modality grouping, stage division within each modality group, and micro-batch allocation within each stage are obtained.

[0204] This invention proposes a pipelined parallel training method for multimodal models on heterogeneous edge devices. It fully utilizes the inherent parallel topology of multimodal models, flexibly combining different modal branches and allocating them to independent device groups. Within each group, a hybrid strategy of pipelined parallelism and intra-stage data parallelism is further employed. Compared to existing methods that suffer from excessive pipeline depth due to linear expansion of the multimodal model, this invention effectively shortens the pipeline depth and reduces the number of micro-batches in transit through modal grouping. This alleviates device memory pressure, reduces pipeline bubble overhead, significantly improves overall training throughput, and reduces overall training time.

[0205] Furthermore, considering the significant differences in computing power and memory capacity among heterogeneous edge devices and the limited communication bandwidth in actual deployment environments, this invention minimizes the bottleneck time of each stage by dynamically optimizing stage partitioning, parallel data configuration within each stage, and non-uniform sub-batch distribution, while satisfying device memory constraints. Simultaneously, this invention considers the communication cost between devices during stage partitioning, avoiding stage segmentation at module boundaries with large output data volumes, and introduces an optional communication compression mechanism to moderately increase computational overhead in exchange for reduced communication data volume. This achieves a balance between computation and communication under low bandwidth conditions, effectively suppressing the constraint of communication bottlenecks on training efficiency.

[0206] In one embodiment, a multimodal model pipelined parallel training system for heterogeneous edge devices is provided, the system comprising:

[0207] The first module is used to implement: construct a multimodal model pipeline parallel training system architecture for heterogeneous edge devices, and with device memory as a constraint and minimizing the overall training time as the objective function, establish an optimization problem of collaborative training in a heterogeneous environment while meeting the model memory requirements;

[0208] The second module is used to: perform performance measurement on the multimodal model on the heterogeneous end-side device, extract the operating characteristics of each module of the multimodal model under different configurations and the state parameters of the heterogeneous end-side device; the state parameters include the available memory of the heterogeneous end-side device and the inter-device communication bandwidth;

[0209] The third module is used to: perform the optimal stage partitioning scheme search for multimodal model hybrid parallelism, use the improved genetic algorithm to perform collaborative grouping of modalities and devices, and combine dynamic programming to determine the stage partitioning within each modal group, and obtain the optimal grouping and stage partitioning scheme.

[0210] The fourth module is used to: allocate devices for the fusion stage according to the optimal grouping and stage division scheme, calculate the overall training time of the optimal scheme, adaptively obtain the optimal training micro-batch, and output a comprehensive parallel training scheme that includes modality grouping, stage division within each modality group, and micro-batch allocation within each stage.

[0211] Specific limitations regarding the parallel training system for multimodal model pipelines for heterogeneous edge devices can be found in the limitations of the parallel training method for multimodal model pipelines for heterogeneous edge devices described above, and will not be repeated here. Each module in the aforementioned parallel training system for multimodal model pipelines for heterogeneous edge devices can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the computer device in hardware form, or stored in the memory of the computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0212] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. The processor executes the computer program to implement the specific steps of the multimodal model pipeline parallel training method for heterogeneous edge devices:

[0213] Step 1: Construct a multimodal model pipeline parallel training system architecture for heterogeneous edge devices, and with device memory as a constraint and minimizing the overall training time as the objective function, establish an optimization problem for collaborative training in a heterogeneous environment while meeting model memory requirements;

[0214] Step 2: Measure the performance of the multimodal model on the heterogeneous edge device, extract the operating characteristics of each module of the multimodal model under different configurations, and the state parameters of the heterogeneous edge device; the state parameters include the available memory and inter-device communication bandwidth of the heterogeneous edge device.

[0215] Step 3: Perform the optimal stage partitioning scheme search for the multimodal model in a hybrid parallel manner. Use the improved genetic algorithm to perform collaborative grouping of modalities and devices, and combine dynamic programming to determine the stage partitioning within each modal group to obtain the optimal grouping and stage partitioning scheme.

[0216] Step 4: Allocate equipment for the fusion stage according to the optimal grouping and stage division scheme, calculate the overall training time of the optimal scheme, adaptively obtain the optimal training micro-batch, and output a comprehensive parallel training scheme that includes modality grouping, stage division within each modality group, and micro-batch allocation within each stage.

[0217] For specific limitations on each step, please refer to the limitations of the multimodal model pipeline parallel training method for heterogeneous edge devices mentioned above, which will not be repeated here.

[0218] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program being implemented when executed by a processor:

[0219] Step 1: Construct a multimodal model pipeline parallel training system architecture for heterogeneous edge devices, and with device memory as a constraint and minimizing the overall training time as the objective function, establish an optimization problem for collaborative training in a heterogeneous environment while meeting model memory requirements;

[0220] Step 2: Measure the performance of the multimodal model on the heterogeneous edge device, extract the operating characteristics of each module of the multimodal model under different configurations, and the state parameters of the heterogeneous edge device; the state parameters include the available memory and inter-device communication bandwidth of the heterogeneous edge device.

[0221] Step 3: Perform the optimal stage partitioning scheme search for the multimodal model in a hybrid parallel manner. Use the improved genetic algorithm to perform collaborative grouping of modalities and devices, and combine dynamic programming to determine the stage partitioning within each modal group to obtain the optimal grouping and stage partitioning scheme.

[0222] Step 4: Allocate equipment for the fusion stage according to the optimal grouping and stage division scheme, calculate the overall training time of the optimal scheme, adaptively obtain the optimal training micro-batch, and output a comprehensive parallel training scheme that includes modality grouping, stage division within each modality group, and micro-batch allocation within each stage.

[0223] For specific limitations on each step, please refer to the limitations of the multimodal model pipeline parallel training method for heterogeneous edge devices mentioned above, which will not be repeated here.

[0224] Exemplary examples are provided to further illustrate the invention in some embodiments.

[0225] This embodiment verifies the effectiveness of the invention in a heterogeneous device environment. This embodiment uses six devices—two NVIDIA Jetson Orin Nanos, one NVIDIA Jetson AGX Orin, one NVIDIA Jetson Xavier NX, one NVIDIA GeForce RTX 3090, and one NVIDIA GeForce RTX 3080—to simulate a heterogeneous edge environment. The effectiveness is compared with four current mainstream pipeline parallelism strategies, specifically including:

[0226] GPP (GraphPipe): Graph Pipeline Model Training, designed for cloud-based homogeneous multimodal model training scenarios. It utilizes the multimodal model topology for training, leveraging intermodal parallelism to reduce training overhead. It divides the model into stages with the goal of minimizing single-sample processing time, and employs data parallelism within each stage to optimize stage processing time.

[0227] Pipedream: A classic pipelined parallel training method designed for cloud-based isomorphic model training scenarios. It linearly expands the model and divides it into stages with the goal of minimizing stage bottleneck time. Within each stage, it uses model replication to reduce stage runtime.

[0228] HPP: Heterogeneous Environment Model Training Method. For heterogeneous environment model training scenarios, the model is linearly expanded and divided into stages with the goal of minimizing the estimated total time. Data parallelism is used within each stage, and micro-batch allocation is performed according to the device's computing power.

[0229] FTPipeHD: For training models in dynamic heterogeneous environments, the model is also linearly expanded. Stages are divided with the goal of minimizing stage bottleneck time. It takes into account the heterogeneity and dynamism of devices and makes dynamic adjustments based on the dynamic changes in device status.

[0230] In heterogeneous edge device scenarios, the training throughput of different algorithms for training CLIP bimodal models is compared under different communication rates. The higher the training throughput, the shorter the overall training time. High communication rate is 300-800GB / s, representing a high communication rate environment in the cloud; low communication rate is 30-80MB / s, representing the communication rate of edge devices; and medium communication bandwidth is the midpoint between high and low communication rates.

[0231] Experimental results as follows Figure 2As shown, in heterogeneous edge scenarios, the training throughput of the proposed method (GADP) and the three heterogeneous perception algorithms HPP and FTPipeHD remains at a high level, meaning the overall training time is shorter and significantly higher than that of the homogeneous algorithms GPP and Pipedream. This is mainly because the homogeneous algorithms are designed for model training in a homogeneous cloud environment, while in heterogeneous edge scenarios, the memory and computing power of devices differ. Using a unified stage division mechanism would cause lower-performance devices to severely drag down the overall training efficiency. Furthermore, the training throughput of the proposed method is also higher than that of the two linear pipeline methods HPP and FTPipeHD. This is because the proposed method achieves modal parallelism, fully utilizing the parallel branch structure of the multimodal model, effectively reducing the pipeline depth, and thus achieving better training results. Similarly, the performance of the graph pipeline method GPP is also better than that of the linear pipeline method Pipedream. On the other hand, the comparative methods have lower training throughput at low communication rates, meaning the overall training time is longer, while the proposed method can still achieve good training results under low communication rate conditions.

[0232] In summary, the multimodal model pipeline parallel training scheme proposed in this invention for heterogeneous edge devices can effectively solve the problems of device heterogeneity, limited memory resources and low communication bandwidth faced in multimodal model training on edge devices, realize parallel training of multimodal models on heterogeneous edge devices with limited memory and communication resources, and optimize the overall training time of the model.

[0233] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A method for pipelined parallel training of multimodal models for heterogeneous edge devices, characterized in that, The method includes the following steps: Step 1: Construct a multimodal model pipeline parallel training system architecture for heterogeneous edge devices, and with device memory as a constraint and minimizing the overall training time as the objective function, establish an optimization problem for collaborative training in a heterogeneous environment while meeting model memory requirements; Step 2: Measure the performance of the multimodal model on the heterogeneous edge device, extract the operating characteristics of each module of the multimodal model under different configurations, and the state parameters of the heterogeneous edge device; the state parameters include the available memory and inter-device communication bandwidth of the heterogeneous edge device. Step 3: Perform the optimal stage partitioning scheme search for the multimodal model in a hybrid parallel manner. Use the improved genetic algorithm to perform collaborative grouping of modalities and devices, and combine dynamic programming to determine the stage partitioning within each modal group to obtain the optimal grouping and stage partitioning scheme. Step 4: Allocate equipment for the fusion stage according to the optimal grouping and stage division scheme, calculate the overall training time of the optimal scheme, adaptively obtain the optimal training micro-batch, and output a comprehensive parallel training scheme that includes modality grouping, stage division within each modality group, and micro-batch allocation within each stage.

2. The method for parallel training of multimodal models in a pipeline for heterogeneous edge devices according to claim 1, characterized in that, In step 1, memory requirements are quantified using a memory model, including: Static memory requirements: Parameters that continuously occupy device memory and whose size remains basically unchanged during training; Dynamic memory requirements: Memory for intermediate activation values ​​generated during the forward propagation process and related to micro-batches and network dimensions; Instantaneous memory requirements: Workspace memory temporarily generated during backpropagation and occupied before being released; Stage memory requirements: determined jointly based on the static memory requirements, dynamic memory requirements, instantaneous memory requirements, and the number of micro-batches in transit in the pipeline; wherein, micro-batches in transit refer to micro-batches that have completed forward propagation but have not completed backward propagation calculations; In step 1, the total training time is quantified using a training time model, including: Training readiness time: determined based on the arrival time of intermediate results from the previous stage; Training start time: determined based on the training readiness time and equipment idle time; Transmission time: determined based on the amount of data transmitted and the transmission rate; when two stages are trained on the same device, the intermediate result transmission time is ignored, i.e., the transmission time is 0. Transmission readiness time: determined based on the completion time of each training phase; Transmission start time: determined based on transmission readiness time and transmission resource idle time; Overall training time: determined based on the completion time of the last micro-batch training in each stage.

3. The method for parallel training of multimodal models in a pipeline for heterogeneous edge devices according to claim 1, characterized in that, The multimodal model pipelined parallel training system architecture for heterogeneous edge devices described in step 1 includes three methods: modal parallelism, pipeline parallelism, and data parallelism; among which, Modal parallelism is adopted between modalities: the modalities of the multimodal model can be independently parallelized. During the training of the multimodal model, different modalities are assigned to different devices for parallel training. During the fusion stage, the outputs of each modality are aligned through a contrastive learning mechanism. The modality adopts a pipelined parallel approach: the layers in a single modality group are divided into multiple stages and distributed to different devices. The different stages of the modality are trained sequentially in a pipeline manner. After the device completes the training of a micro-batch, the intermediate results are transmitted to the next stage while the next micro-batch of the current stage is trained in parallel. The data parallelism approach is adopted within the phase: within the modal phase, the input micro-batch is split and distributed to multiple devices for training of their respective batches, and gradient synchronization is performed after a complete batch is trained.

4. The method for parallel training of multimodal models in a pipeline for heterogeneous edge devices according to claim 1, characterized in that, The optimization problem of collaboratively training a multimodal model using heterogeneous end-devices in step 1 is expressed as follows: constraint Due to memory constraints, it is necessary to ensure that the memory requirements allocated by the device during a given period do not exceed the available memory of the device; constraint The phase division constraint ensures that each module of the modality is assigned to a specific phase, and that there is no overlap between phases; Constraints are assigned to devices, requiring at least one device to be allocated for each stage to support stage training; constraint Each stage is required to be responsible for training only one stage, in order to avoid stages being trained on the same device, which would affect the efficiency of pipelined parallel training. In the formula, Minimize the total training time , For equipment The allocated stage memory requirements, For equipment Available memory, Represents the k-th mode The Middle Phase Corresponding distribution device set The Middle One device ; Indicates the total number of stages. Represent each mode An ordered set of modules Representing the k-th mode respectively The first in The first stage, the first The number of stages; K represents the total number of modes. Indicates device Whether to be assigned to a stage Equipment collection .

5. The method for parallel training of multimodal models in a pipeline for heterogeneous edge devices according to claim 1, characterized in that, Step 2 extracts runtime features, specifically including: On different devices, each module of the multimodal model is simulated to generate adapted inputs according to the input specifications. The forward and backward time, input and output data volume, parameter scale and activation value memory usage of each module of the target multimodal model are measured and recorded on different devices as the micro-batch b changes. Calculate the memory requirements at each level under different micro-batch sizes.

6. The method for parallel training of multimodal models in a pipeline for heterogeneous edge devices according to claim 1, characterized in that, Step 3 specifically includes: An improved genetic algorithm is used for modal grouping and heterogeneous device allocation for multimodal applications; For the selectable micro-batch size, modal grouping and device grouping are jointly genetically encoded. After standardization to eliminate equivalent redundant solutions, feasibility repair is performed based on device memory constraints. The dynamic programming method is used to divide the chromosome individual into stages and calculate the training time, and the chromosome fitness is evaluated. At the same time, a fitness cache is established to reuse the evaluation results of the same standardized individuals. Through continuous iterative evolution via selection, crossover, and mutation operations, the optimal grouping scheme for overall training time and its corresponding stage division results are generated for each candidate micro-batch.

7. The method for parallel training of multimodal models in a pipeline for heterogeneous edge devices according to claim 6, characterized in that, In step 3, joint gene encoding is performed on modality grouping and device grouping. The specific process includes: A chromosome with a two-layer hybrid coding structure is constructed, the total number of gene loci being the sum of the number of modalities and the number of devices; the front segment of the chromosome represents the modal grouping scheme, each bit in the front segment corresponds to one modality, and the value represents the modal group number to which the modality is assigned; the back segment of the chromosome represents the device grouping scheme, each bit in the back segment corresponds to one device, and the value represents the modal group number to which the device is assigned. In step 3, standardization eliminates equivalent redundant solutions. The specific process includes: The modal grouping encoding of the first segment of the chromosome is relabeled according to the order of first appearance to obtain a canonical representation. Based on this, the device grouping encoding of the second segment is uniformly mapped and invalid pointers are corrected. After integrity verification, device reassignment is performed on the modal group of unassigned devices to meet the constraints, thereby merging multiple equivalent codes into a unique canonical code to reduce search space redundancy. Step 3 involves performing feasibility repairs based on device memory constraints, specifically including: Based on the device grouping results, calculate the memory supply and demand of each modal group, and mark the modal group with insufficient supply (i.e., supply less than the preset threshold) as the memory-insufficient group, and the other group as the memory-sufficient group. Migrate devices from modal groups with ample memory and more than one device to groups with insufficient memory and update supply and demand values. Repeat the current process until all modal groups meet memory constraints or reach the preset repair limit. If there are still modal groups that have not been allocated devices, perform fallback allocation while keeping the remaining modal groups feasible to ensure that each effective modal group corresponds to at least one device and obtain feasible chromosome codes that meet memory constraints.

8. The method for parallel training of multimodal models in a pipeline for heterogeneous edge devices according to claim 6, characterized in that, In step 3, the criteria for dividing the stages in dynamic programming are as follows: For each modality group, a dynamic programming recursion is constructed with the goal of minimizing the pipeline bottleneck time. During the recursion process, if multiple candidate splitting schemes have the same bottleneck time, the scheme with the smaller stage delay is further selected. After each modality group has completed stage division, the total training time is calculated, and this total training time is used as the fitness value of the corresponding chromosome individual. The bottleneck time is the larger value between the intra-stage computation time and the inter-stage communication time, and the stage delay is the sum of the computation time and the bottleneck time.

9. The method for parallel training of multimodal models in a pipeline for heterogeneous edge devices according to claim 8, characterized in that, In step 3, the dynamic programming stage division also includes data parallelism and micro-batch adjustment mechanisms, as well as communication compression mechanisms; The data parallelism and micro-batch adjustment mechanism allows continuous module intervals to be deployed to multiple devices to form data parallel subgroups within each candidate stage. The total amount of micro-batch in the stage is non-uniformly divided into sub-micro-batch according to the differences in device computing power, and memory feasibility is verified. The communication compression mechanism states that when the stage computation time is less than the stage communication time, the compression algorithm library is traversed to select the compression method that minimizes the stage bottleneck time in order to balance computation and communication overhead.

10. The method for parallel training of multimodal models in a pipeline for heterogeneous edge devices according to claim 1, characterized in that, Step 4 involves the allocation of equipment during the fusion phase, specifically including: After the modality phase is divided, the device containing the last phase of the modality that satisfies memory constraints and has the longest training time for a single micro-batch is selected, and the multimodal fusion phase is deployed.

Images