Distributed training method of multi-modal model, electronic device and storage medium

Abstract

The application relates to the technical field of artificial intelligence, and provides a distributed training method of a multi-modal model, an electronic device and a storage medium.The method comprises the following steps: acquiring modal characteristic indexes of each mode in the multi-modal model, the modal characteristic indexes of each mode being used for representing the calculation characteristics, data characteristics and communication characteristics of each mode; calculating training resource requirements of each mode according to the modal characteristic indexes of each mode; differentially determining a parallel strategy for each mode based on the training resource requirements of each mode; and executing a training process of the multi-modal model in a distributed computing environment based on the parallel strategy of each mode.The application can accurately match the dynamic and heterogeneous resource requirements of each mode in the training process, thereby effectively solving the problems of idle or bottleneck hardware resources caused by the mismatch between the strategy and the resources, avoiding resource waste, and significantly improving the overall training efficiency and resource utilization rate of the multi-modal large model in the distributed environment.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a distributed training method, electronic device, and storage medium for a multimodal model. Background Technology

[0002] Current technologies for distributed training of multimodal models typically employ a single, fixed parallel strategy for the entire model before training. This might involve splitting the model across different computing nodes using a data parallelism strategy and synchronizing parameters using a fixed communication method. However, the computational characteristics and resource requirements of different modalities (such as text, image, and audio) in a multimodal model change dynamically during training. This fixed parallel strategy cannot adapt to the heterogeneity and dynamic needs of each modality, easily leading to idle resources on some computing nodes or resource bottlenecks on others, resulting in low overall training efficiency and resource utilization. Summary of the Invention

[0003] This invention provides a distributed training method, electronic device, and storage medium for multimodal models to address the deficiencies in related technologies.

[0004] This invention provides a distributed training method for a multimodal model, comprising the following steps:

[0005] Obtain modal characteristic indices for each mode in the multimodal model. These indices are used to characterize the computational, data, and communication characteristics of each mode.

[0006] Calculate the training resource requirements for each modality based on its modal characteristic indices;

[0007] Based on the training resource requirements of each modality, a parallel strategy is determined differently for each modality.

[0008] The training process of the multimodal model is executed in a distributed computing environment based on the parallel strategy of each modality.

[0009] According to the distributed training method for a multimodal model provided by the present invention, the modal characteristic indicators of each modality include computational characteristic indicators, data characteristic indicators, and communication characteristic indicators.

[0010] The step of calculating the training resource requirements for each modality based on its modal characteristic indices includes:

[0011] Based on the computational characteristic indices of each mode, determine the parameter quantities of each mode;

[0012] Based on the data characteristic indicators of each modality, determine the ratio between the data volume of other modalities and the data volume of each modality;

[0013] Based on the communication characteristic indicators of each mode, determine the maximum bandwidth and current bandwidth of the environment in which each mode is located;

[0014] The training resource requirements for each modality are calculated based on the number of parameters for each modality, the ratio between the amount of data for other modalities and the amount of data for each modality, and the difference between the maximum bandwidth and the current bandwidth of the environment in which each modality is located.

[0015] According to a distributed training method for a multimodal model provided by the present invention, the step of calculating the training resource requirements of each modality based on the number of parameters of each modality, the ratio between the amount of data of other modalities and the amount of data of each modality, and the difference between the maximum bandwidth and the current bandwidth of the environment in which each modality is located, includes:

[0016] The training resource requirements for each modality are obtained by weighting and summing the logarithms of the parameters of each modality, the ratio between the data volume of other modalities and the data volume of each modality, and the difference between the maximum bandwidth of the environment in which each modality is located and the current bandwidth.

[0017] According to a distributed training method for a multimodal model provided by the present invention, the step of determining a parallel strategy differentiated for each modality based on the training resource requirements of each modality includes:

[0018] If the training resource requirement of any modality exceeds the first threshold, then the corresponding modality is determined to adopt a parallel strategy that combines tensor parallelism and model parallelism.

[0019] If the training resource requirement of any modality is less than or equal to the first threshold, then the corresponding modality is determined to adopt a data parallel strategy.

[0020] According to a distributed training method for a multimodal model provided by the present invention, if the training resource requirement of any modality is less than or equal to a first threshold, then determining that the corresponding modality adopts a data parallel strategy includes:

[0021] If the training resource requirement of any modality is less than or equal to the first threshold and the memory utilization is greater than the second threshold, then the corresponding modality is determined to adopt a parallel strategy that combines data parallelism and gradient compression.

[0022] If the training resource requirement of any modality is less than or equal to the first threshold and the modality heterogeneity is greater than the third threshold, then the corresponding modality is determined to adopt a parallel strategy that combines pipeline parallelism and priority scheduling.

[0023] According to a distributed training method for a multimodal model provided by the present invention, the step of obtaining modal characteristic indices for each modality in the multimodal model includes:

[0024] Dynamic code analysis is performed on the multimodal model to statistically analyze the computational characteristic indicators of each modality, including the number of floating-point operations per second, the number of parameters, and the proportion of computationally intensive layers.

[0025] The training data of the multimodal model is pre-scanned, and the data characteristic indicators of each modality are statistically analyzed. The data characteristic indicators include single sample data size, batch size range, and data loading time.

[0026] The simulation training of the multimodal model is performed, and the communication characteristic indicators of each modality are recorded. The communication characteristic indicators include the communication synchronization frequency, the amount of cross-modal interaction data, and the degree of communication impact for each training step.

[0027] According to a distributed training method for a multimodal model provided by the present invention, the training process of the multimodal model in a distributed computing environment based on a parallel strategy for each modality includes:

[0028] The parallel strategies for each modality are integrated into the training execution engine, and the training process of the multimodal model is executed in a distributed computing environment according to the parallel strategies for each modality.

[0029] According to a distributed training method for a multimodal model provided by the present invention, the process of training the multimodal model in a distributed computing environment further includes:

[0030] By deploying a monitoring agent in the distributed computing environment, resource usage information for each modality is collected in real time.

[0031] When the training duration reaches the preset duration or when the resource usage information collected in real time triggers the preset conditions, the process returns to the steps of obtaining modal characteristic indicators, calculating training resource requirements, determining parallel strategies, and training models.

[0032] The present invention also provides a distributed training device for a multimodal model, comprising the following units:

[0033] The acquisition unit is used to acquire the modal characteristic indices of each mode in the multimodal model. The modal characteristic indices of each mode are used to characterize the computational characteristics, data characteristics and communication characteristics of each mode.

[0034] The computing unit is used to calculate the training resource requirements of each modality based on the modal characteristic indicators of each modality.

[0035] The determination unit is used to determine the parallel strategy differently for each modality based on the training resource requirements of each modality;

[0036] The training unit is used to execute the training process of multimodal models in a distributed computing environment based on parallel strategies for each modality.

[0037] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement a distributed training method for a multimodal model as described above.

[0038] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a distributed training method for a multimodal model as described above.

[0039] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements a distributed training method for a multimodal model as described above.

[0040] The distributed training method, electronic device, and storage medium for multimodal models provided by this invention first acquires and quantifies the computational, data, and communication characteristics of each modality. Then, based on these characteristics, it calculates the differentiated training resource requirements of each modality. Finally, based on these requirements, it independently determines the optimal parallel strategy for each modality, achieving personalized and differentiated configuration of parallel methods for different modalities. Because this invention no longer employs the fixed parallel strategy of traditional technologies, but can accurately match the dynamic and heterogeneous resource requirements of each modality during training, it effectively solves the problem of idle or bottleneck hardware resources caused by mismatch between strategy and resources, avoids resource waste, and significantly improves the overall training efficiency and resource utilization of large multimodal models in a distributed environment. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in this invention or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a flowchart illustrating the distributed training method for multimodal models provided by the present invention.

[0043] Figure 2 This is a flowchart illustrating another distributed training method for multimodal models provided by the present invention.

[0044] Figure 3 This is a schematic diagram of the structure of the distributed training device for the multimodal model provided by the present invention.

[0045] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0047] With the rapid development of artificial intelligence technology, large-scale multimodal pre-trained models (such as CLIP, LLAVA, and other models with hundreds of billions / trillions of parameters that integrate text, images, audio, and video) have shown enormous application potential in fields such as computer vision and natural language processing. The training of these models typically relies on distributed computing clusters. Especially in cloud computing environments, heterogeneous computing nodes (GPU / CPU / TPU hybrid architecture), and cross-data center scenarios, how to efficiently perform distributed parallel training has become a key technical challenge.

[0048] When performing distributed training of multimodal models, related technologies typically employ the following process: First, the multimodal model (including components such as text encoders, image encoders, and cross-modal fusion layers) is divided into multiple subtasks according to a fixed parallel strategy and distributed to different computing nodes. Second, the multimodal training data (text, image, and audio samples) is distributed to each node in a uniform batch size, and each node independently performs forward and backward propagation computations. Then, a fixed communication strategy (such as All-Reduce full gradient synchronization) is used to complete parameter updates across nodes, with the cross-modal fusion layer using the same parallel logic as the single-modal layer. Finally, computing resources (such as GPU memory and communication bandwidth) are allocated according to a statically preset ratio, without distinguishing the characteristic differences of different modalities.

[0049] In other words, the parallel strategies employed by the relevant technologies are determined and remain fixed before training begins, and cannot be dynamically adjusted according to the actual situation during training. The computational characteristics of different modalities (such as text modality and image modality) in multimodal models vary significantly, and fixed parallel strategies cannot optimize for the heterogeneity of each modality, which can easily lead to idle resources or resource bottlenecks on some computing nodes, resulting in hardware resource redundancy and waste.

[0050] Secondly, the relevant technologies allocate computing resources according to a static preset ratio, without considering the dynamic changes in resource requirements of different modalities during training. For example, the image modality requires high memory to load the original image in the early stage of training (such as the data loading stage), but requires high computing power for computation in the middle stage of training (such as the feature extraction stage). The static resource allocation method is prone to memory overflow or idle computing power.

[0051] In addition, different hardware clusters (such as 8-card V100 and 32-card A100) have huge differences in terms of video memory, bandwidth and computing power. Manually formulated static strategies are difficult to take into account all application scenarios, ultimately resulting in low overall training efficiency and insufficient resource utilization.

[0052] To address this, the present invention provides a distributed training method for multimodal models, which aims to dynamically adjust the parallel strategy and resource allocation scheme according to the computational characteristics and resource requirements of different modalities. This solves the technical problems of low resource utilization and low training efficiency caused by fixed parallel strategies in related technologies, and achieves efficient resource utilization and optimized training performance in distributed training of multimodal models.

[0053] in, Figure 1 This is a flowchart illustrating the distributed training method for multimodal models provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130 and 140.

[0054] Step 110: Obtain the modal characteristic indices of each mode in the multimodal model. The modal characteristic indices of each mode are used to characterize the computational characteristics, data characteristics and communication characteristics of each mode.

[0055] Here, a multimodal model can be understood as a deep learning model that integrates multiple data modalities (such as text, images, audio, etc.) for joint training. It can be a large-scale pre-trained model such as CLIP or LLAVA. A multimodal model typically includes multiple modal encoders (such as text encoders, image encoders, audio encoders, etc.) and cross-modal fusion layers.

[0056] Modal characteristic indices for each modality are used to characterize the computational, data, and communication characteristics of each modality. They are also used to quantify the differences in characteristics of each modality during training, reflecting the characteristics of different modalities in terms of computational complexity, data processing requirements, and communication overhead.

[0057] The computational characteristics are used to characterize the computational complexity and computational resource requirements of each modality during the training process. For example, the image modality typically involves a large number of convolution operations and matrix multiplication operations, so its computational characteristics are high computational density; the text modality mainly involves sequence processing and attention mechanism computation, so its computational characteristics are that the computational complexity is closely related to the length of the input sequence, and the computational load increases dramatically when processing long sequences.

[0058] Data characteristics are used to characterize the scale, dimensionality, sampling characteristics, and other data-level features of the training data for each modality. For example, the raw data of the image modality is usually a high-dimensional tensor, so its data characteristics are that data loading and preprocessing require high memory bandwidth; the data of the text modality is usually a variable-length sequence, which needs to be padded or truncated, so its data characteristics are that its memory usage is relatively small.

[0059] Furthermore, the data characteristics of different modalities differ at different training stages. For example, for the image modality, high memory is required in the early stages of training to load the original image data, potentially reaching tens of gigabytes. However, in the middle stages of training, the data has been loaded into the GPU memory, at which point high computing power is mainly needed for forward and backward propagation calculations. For the text modality, its data characteristics are relatively stable across different training stages.

[0060] Communication characteristics are used to characterize the communication requirements and overhead of each modality during distributed training. For example, for a cross-modal fusion layer, frequent data exchange and parameter synchronization are required between computing nodes of different modalities, so its communication characteristics are communication-intensive and sensitive to network latency. For a single-modal encoder (such as a text encoder, image encoder, audio encoder, etc.), it can complete the computation relatively independently on its own computing node, so its communication characteristics are relatively low communication overhead, mainly concentrated on parameter synchronization within the modality.

[0061] Since different modes have significant differences in computational characteristics, data characteristics, and communication characteristics, these differences can be quantitatively characterized by obtaining modal characteristic indicators for each mode, thereby providing a basis for subsequent resource requirement calculation and parallel strategy determination.

[0062] Step 120: Calculate the training resource requirements for each modality based on its modal characteristic indices.

[0063] Specifically, training resource requirements refer to the quantitative requirements of computing resources, storage resources, communication resources, etc., needed by each modality during distributed training. Training resource requirements may include, but are not limited to, computing resource requirements (such as GPU computing power, CPU computing power), storage resource requirements (such as GPU video memory, host memory), and communication resource requirements (such as network bandwidth, communication latency tolerance), etc.

[0064] As an optional embodiment, the training resource requirements for each modality can be quantified using a resource requirement calculation model based on the modal characteristic indicators of each modality obtained in step 110. The resource requirement calculation model can be a calculation model based on empirical formulas, a prediction model based on performance models, or a statistical model based on historical data, etc.

[0065] As another optional embodiment, the computational resource requirements of each modality can be calculated based on computational characteristics, the storage resource requirements of each modality can be calculated based on data characteristics, and the communication resource requirements of each modality can be calculated based on communication characteristic indicators. Then, the computational resource requirements, storage resource requirements, and communication resource requirements are summarized to obtain the training resource requirements of each modality.

[0066] Specifically, computational resource requirements characterize the floating-point arithmetic capabilities needed to complete a training step; for example, the computation time required to complete one forward and backward propagation cycle can be used to measure computational resource requirements. Storage resource requirements characterize the memory space required to store model parameters, activation values, gradients, and optimizer states during training; for example, peak GPU memory usage can be used to measure storage resource requirements. Communication resource requirements characterize the network capabilities required to synchronize parameters or exchange data between nodes; for example, the minimum required network bandwidth or the maximum tolerable communication latency can be used to measure communication resource requirements.

[0067] It should be noted that training resource requirements are not static values, but may change dynamically with the training stage. Therefore, this embodiment can calculate the training resource requirements corresponding to different training stages (such as the initial loading stage, the intermediate feature extraction stage, and the late fine-tuning stage), and then dynamically adjust resource allocation and parallel strategies according to the training resource requirements of different training stages. For example, for text modalities, small sample sizes and small learning rates are typically used for training in the fine-tuning stage, which may reduce the parameter update frequency or the number of updated parameters, resulting in a significant drop in GPU utilization from 70% in regular training to 30%, and a corresponding decrease in computing power requirements. However, in the pre-training stage, large-scale datasets need to be processed and frequent gradient updates are performed, which increases the computing power requirements accordingly.

[0068] Step 130: Based on the training resource requirements of each modality, determine the parallel strategy for each modality in a differentiated manner.

[0069] Specifically, determining the parallel strategy in a differentiated manner can be understood as no longer using a single, fixed parallel mode for the entire multimodal model, but rather selecting the most suitable combination of parallel techniques independently and specifically for each modality or combination of modalities based on the training resource requirements of different modalities calculated in step 120.

[0070] Parallelism strategies can include data parallelism, tensor parallelism, pipeline parallelism, ZeRO (Zero Redundancy Optimizer) strategy, and gradient splitting strategy, etc.

[0071] Among them, the data parallel strategy refers to completely copying the model to multiple computing devices, with each device processing a portion of the training data, and finally updating the model by aggregating gradients through communication. For example, a batch of data can be evenly distributed to 4 GPUs, and each GPU can independently calculate the gradient and then perform an All-Reduce operation.

[0072] Tensor parallelism refers to splitting a single large operator in the model across multiple devices for parallel computation. For example, the weight matrix of a fully connected layer can be split column-wise across two GPUs, with each GPU only computing a portion of the results, which are then merged through communication.

[0073] The pipelined parallel strategy refers to distributing different layers of a model to different devices to form a computational pipeline. For example, in a 12-layer model, the first 4 layers are executed on GPU0, the middle 4 layers on GPU1, and the last 4 layers on GPU2.

[0074] The ZeRO strategy eliminates storage redundancy by splitting model parameters, gradients, and optimizer states across data-parallel devices. For example, the ZeRO-3 strategy splits all three, so that each GPU only needs to store a portion of the model state, greatly reducing the memory requirements of a single GPU.

[0075] The gradient splitting strategy refers to splitting the gradient after it is generated and storing it on different devices instead of having each device hold a complete copy of the gradient. The gradient is then collected when the parameters are updated.

[0076] For example, if a modality's storage resource requirements are extremely high, exceeding the GPU memory capacity of a single computing node, then a tensor parallel strategy or a ZeRO strategy capable of partitioning the model state should be chosen. If a modality's computational resource requirements are extremely high, causing excessive computation time on a single node and becoming a bottleneck in the entire training process, then a tensor parallel strategy or a data parallel strategy can be chosen. If there are obvious serial dependencies within or between modalities, then a pipelined parallel strategy can be chosen. If a modality has a massive amount of training data, then a data parallel strategy can be chosen.

[0077] Step 140: Based on the parallel strategy of each modality, execute the training process of the multimodal model in a distributed computing environment.

[0078] Specifically, a distributed computing environment can be understood as a computing system composed of interconnected computing nodes. These computing nodes can be physical servers, virtual machines, or containers. Each node can contain one or more computing units, such as a graphics processing unit (GPU), a central processing unit (CPU), or a tensor processing unit (TPU).

[0079] The process of training a multimodal model refers to instantiating the differentiated parallel strategies determined for each modality in step 130 and driving the entire model to complete the complete training iteration from data loading, forward propagation, back propagation to parameter update in a distributed environment.

[0080] As an optional implementation, the computation graph of the multimodal model can be partitioned according to a determined parallel strategy. For example, suppose the multimodal model contains an image encoder, a text encoder, and a cross-modal fusion layer. If step 130 determines that the image encoder uses a tensor parallel strategy and a ZeRO strategy, then the weight parameters, gradients, and optimizer states within the encoder are partitioned onto a specified set of GPUs. If the text encoder is determined to use a data parallel strategy, then a complete copy of the text encoder is copied to another set of GPUs.

[0081] Next, based on the resource requirements calculated in step 120, the underlying resource scheduler allocates and binds specific hardware resources to the processes executing each modality training task. For example, a higher memory quota is allocated to the GPU carrying the image modality, and computationally demanding tasks such as image feature extraction are prioritized for scheduling on the more powerful GPU, while other tasks are scheduled on the less powerful GPU, thus achieving reasonable utilization of heterogeneous resources. After the above hardware resource deployment and binding are completed, the training task begins to be executed. During the training iteration, the computational tasks of each modality are executed concurrently on their allocated resources.

[0082] The distributed training method for multimodal models provided in this embodiment first acquires and quantifies the computational, data, and communication characteristics of each modality. Then, based on these characteristics, it calculates the differentiated training resource requirements for each modality. Finally, based on these requirements, it independently determines the optimal parallel strategy for each modality, achieving personalized and differentiated configuration of parallel methods for different modalities. Because this embodiment no longer employs the fixed parallel strategy of traditional techniques, but rather accurately matches the dynamic and heterogeneous resource requirements of each modality during training, it effectively solves the problem of idle or bottleneck hardware resources caused by mismatch between strategy and resources, avoids resource waste, and significantly improves the overall training efficiency and resource utilization of large multimodal models in a distributed environment.

[0083] It should be noted that the execution entity in this embodiment can be a distributed training platform, a software framework integrating the method of this embodiment, or a cloud service that manages a distributed computing cluster. This execution entity runs on a distributed computing system composed of multiple computing nodes.

[0084] Taking the training of image-text multimodal models (such as CLIP and LLAVA) as an example, the method in this embodiment analyzes the modal characteristics of the image encoder and the text encoder. Based on the analysis results, tensor parallelism and ZeRO strategies are determined for the image encoder, which has high memory and computational requirements, while data parallelism strategies are determined for the text encoder. By implementing this differentiated parallel scheme, the model can be trained efficiently, enabling it to have powerful capabilities such as cross-modal retrieval, visual question answering, or image description generation.

[0085] Taking the training of multimodal models such as video-audio-text as an example, the method in this embodiment performs characteristic analysis on the processing modules of video streams, audio streams, and text streams respectively. Due to its spatiotemporal complexity, video modality may require pipeline parallelism combined with tensor parallelism; audio modality may be suitable for data parallelism; and text modality may also be suitable for data parallelism. This method, by independently configuring the optimal parallel strategy for each modality and properly handling their data interactions at the fusion layer, can efficiently train complex models for video content understanding, automatic caption generation, or multimodal source sentiment analysis.

[0086] Furthermore, the execution entity of this embodiment, namely the distributed computing system, typically consists of multiple computing nodes containing AI chips such as GPUs (Graphics Processing Units) and TPUs (Tensor Processing Units). This embodiment, by dynamically matching optimal parallel strategies (such as data parallelism, tensor parallelism, pipelined parallelism, etc.) for different modalities, can maximize the computing potential and memory capacity of these AI chips, avoiding resource bottlenecks and hardware idleness caused by inappropriate strategies. This enables efficient collaborative work across the entire heterogeneous distributed cluster, significantly accelerating the training process of large-scale multimodal models.

[0087] Based on the above embodiments, the modal characteristic indicators of each mode include computational characteristic indicators, data characteristic indicators, and communication characteristic indicators.

[0088] Based on the modal characteristic indices of each modality, the training resource requirements for each modality are calculated, including:

[0089] Based on the computational characteristic indices of each mode, determine the parameter quantities of each mode;

[0090] Based on the data characteristic indicators of each modality, determine the ratio between the data volume of other modalities and the data volume of each modality;

[0091] Based on the communication characteristic indicators of each mode, determine the maximum bandwidth and current bandwidth of the environment in which each mode is located;

[0092] The training resource requirements for each modality are calculated based on the number of parameters for each modality, the ratio between the amount of data for other modalities and the amount of data for each modality, and the difference between the maximum bandwidth and the current bandwidth of the environment in which each modality is located.

[0093] Specifically, the modal characteristic indicators for each mode include computational characteristic indicators, data characteristic indicators, and communication characteristic indicators. Table 1 is a list of characteristic indicators for each mode provided by this invention.

[0094] Table 1. List of characteristic indices for each mode

[0095]

[0096] Computational performance metrics are metrics used to quantify the computational workload and complexity of each modality during training. These metrics may include the number of floating point operations per second (FLOPS) performed by each modality encoder, the number of parameters (params), and the proportion of computationally intensive layers (such as the attention layer accounting for 60% of the computational workload of the text modality).

[0097] Since the computational characteristic indicators of each modality quantify the intrinsic computational complexity and model size of each modality, the number of parameters for each modality can be analyzed based on these indicators. The number of parameters for each modality refers to the total number of all trainable parameters in the corresponding sub-network (such as the image encoder), directly reflecting the model's size and complexity. For example, through static analysis of the structure of a multimodal model, it can be found that the number of parameters in its image encoder is 307M (millions), while the number of parameters in its text encoder is 110M.

[0098] Data characteristic metrics refer to the measurement parameters used to quantify the scale, structure and processing requirements of training data for each modality. These can include the size per sample, batch size range, and load latency for each modality.

[0099] Since the data characteristic indicators of each modality quantify the amount of data processed by each modality, the unit sample data size of each modality can be analyzed based on these indicators, yielding the ratio between the data volume of other modalities and the data volume of each modality. Here, the ratio between the data volume of other modalities and the data volume of each modality is a relative value used to measure the proportion of the data load required to be processed by the current modality relative to that of other modalities in a single training iteration.

[0100] For example, in a multimodal training task involving text and images, suppose the preprocessed image sample is 2MB in size, while the corresponding text sample is 2KB in size. Then, for the image modality, the ratio (text data size / image data size) is 2KB / 2MB = 1 / 1024; for the text modality, the ratio is 2MB / 2KB = 1024. The difference between these two ratios (1 / 1024 is much smaller than 1024) shows that the image modality requires significantly more memory and bandwidth during the data loading phase than the text modality.

[0101] Communication characteristic metrics refer to the measurement parameters used to quantify the frequency, data volume, and performance requirements of inter-node interactions in distributed training for each modality. These can include the communication synchronization frequency (Sync freq) for each training step of each modality, the cross-modal data volume, and the degree of communication impact (Sensitivity).

[0102] Since the communication characteristic indicators of each modality clearly define the dependence of each modality on network resources and the communication mode, the network link status between computing nodes carrying the training tasks of each modality can be analyzed based on these indicators, yielding the current bandwidth and maximum bandwidth of the environment in which each modality resides. Here, the maximum bandwidth refers to the theoretical peak bandwidth of the network link between computing nodes carrying the training task of that modality, while the current bandwidth refers to the actual available bandwidth detected in real time by network monitoring tools during the training process. The difference between the maximum bandwidth and the current bandwidth reflects the current network link load level and congestion degree. The larger the difference, the higher the current network load, the more congested the link, and the more scarce the effective bandwidth resources available for that modality; the smaller the difference, the lower the current network load, the smoother the link, and the more abundant the effective bandwidth resources available for that modality.

[0103] Finally, based on the number of parameters in each modality, the ratio between the data volume of other modalities and the data volume of each modality, and the difference between the maximum bandwidth and the current bandwidth of the environment in which each modality is located, the training resource requirements of each modality are calculated. These training resource requirements integrate the static complexity of the model itself, the relative data load between different modalities, and the dynamic network state of the distributed environment, so that the calculated resource requirement assessment results are more comprehensive and accurate, and can dynamically reflect the real resource consumption during the training process.

[0104] As an optional embodiment, the training resource requirements for each modality can be obtained by weighting and summing the parameter quantities of each modality, the ratio between the data quantities of other modalities and the data quantities of each modality, and the difference between the maximum bandwidth of the environment in which each modality is located and the current bandwidth.

[0105] Based on any of the above embodiments, the training resource requirements for each modality are calculated based on the number of parameters for each modality, the ratio between the amount of data from other modalities and the amount of data from each modality, and the difference between the maximum bandwidth and the current bandwidth of the environment in which each modality resides. This includes:

[0106] The training resource requirements for each modality are obtained by weighting and summing the logarithms of the parameters of each modality, the ratio between the data volume of other modalities and the data volume of each modality, and the difference between the maximum bandwidth of the environment in which each modality is located and the current bandwidth.

[0107] Specifically, the logarithm of the number of parameters for each modality refers to taking the logarithm of the total number of trainable parameters in the model components included in each modality. Considering that the number of parameters in different modalities may differ by orders of magnitude, directly using the number of parameters for calculation might lead to this indicator having too much weight in the final resource requirement calculation, thus masking the influence of other important factors such as data characteristics and communication environment. Therefore, when calculating the training resource requirements for each modality, the logarithm of the number of parameters for each modality is used to scale and normalize this indicator, making its influence smoother and thus more balanced in considering the characteristics of each dimension.

[0108] Taking a multimodal model that includes text and image modalities as an example, the training resource requirements for the text modality can be calculated based on the following formula:

[0109] R_txt=αlog(P_txt)+β(D_img / D_txt)+γ(B_max-B_curr)

[0110] Where R_txt represents the training resource requirements of the text modality, P_txt represents the number of parameters of the text modality, D_img represents the amount of image modality data, D_txt represents the amount of text modality data, B_max represents the maximum bandwidth, B_curr represents the current bandwidth, and α, β and γ are weighting coefficients that can be obtained by regression from historical training data.

[0111] Taking a multimodal model that includes text, image, and audio modalities as an example, the training resource requirements for the image modality can be calculated based on the following formula:

[0112] R_img=αlog(P_img)+β(D_txt / D_img+D_audio / D_img)+γ(B_max-B_curr)

[0113] Where R_img represents the training resource requirements of the image modality, P_img represents the number of parameters of the image modality, D_img represents the amount of image modality data, D_audio represents the amount of audio modality data, D_txt represents the amount of text modality data, B_max represents the maximum bandwidth, B_curr represents the current bandwidth, and α, β and γ are weighting coefficients that can be obtained by regression from historical training data.

[0114] Based on any of the above embodiments, and based on the training resource requirements of each modality, a parallel strategy is determined differently for each modality, including:

[0115] If the training resource requirement of any modality exceeds the first threshold, then the corresponding modality is determined to adopt a parallel strategy that combines tensor parallelism and model parallelism.

[0116] If the training resource requirement for any modality is less than or equal to the first threshold, then the corresponding modality is determined to adopt a data parallel strategy.

[0117] Specifically, the first threshold can be understood as a preset resource upper limit benchmark, which is used to determine whether a single computing node can independently carry out a training task of a modality. This threshold can be determined based on the standard hardware configuration of computing nodes in a distributed computing environment.

[0118] Furthermore, the first threshold can be a multi-dimensional vector, corresponding to the critical values ​​of computing resources, storage resources, and communication resources respectively, or it can be a comprehensive score calculated based on the weighted average of these resources. This embodiment does not specifically limit it in this way.

[0119] After determining the first threshold, this embodiment compares the training resource requirements of each modality with the first threshold and decides on a parallel strategy based on the comparison results.

[0120] If the training resource requirement of any modality exceeds the first threshold, it indicates that the number of parameters of that modality is too large, or that the intermediate activation values ​​generated during its forward propagation are too large, causing its storage resource requirement to exceed the upper limit of the GPU memory of a single computing node.

[0121] At this point, a single parallel strategy (such as pure data parallelism) is no longer sufficient to solve the problem. Therefore, this embodiment determines that a parallel strategy combining tensor parallelism and model parallelism is adopted for the corresponding modality.

[0122] Tensor parallelism involves horizontally partitioning a large single operator across multiple computing nodes, allowing each node to store and compute only a portion of the model, thus directly reducing peak memory usage and computational load. Model parallelism distributes different layers of the model vertically across different computing nodes, forming a computational pipeline, further partitioning the model and reducing the storage burden on individual nodes. By combining these two strategies, modalities that cannot be accommodated by any single computing node can be effectively partitioned and deployed across multiple computing nodes for collaborative training.

[0123] If the training resource requirement of any modality is less than or equal to the first threshold, it indicates that the modality can be fully loaded into the GPU memory of a single computing node, and its computational load is within the capacity of that single computing node. In this case, the bottleneck for training is no longer the model size, but rather how to process massive amounts of training data faster to improve overall training throughput. Based on this, a data-parallel strategy is adopted for this modality. Data parallelism involves placing a complete copy of the model on multiple computing nodes and distributing each batch of training data, after splitting it, to these computing nodes, allowing them to process different data in parallel.

[0124] Based on any of the above embodiments, if the training resource requirement of any modality is less than or equal to a first threshold, then it is determined that the corresponding modality adopts a data parallel strategy, including:

[0125] If the training resource requirement of any modality is less than or equal to the first threshold and the memory utilization is greater than the second threshold, then the corresponding modality is determined to adopt a parallel strategy that combines data parallelism and gradient compression.

[0126] If the training resource requirement of any modality is less than or equal to the first threshold and the modality heterogeneity is greater than the third threshold, then the corresponding modality is determined to adopt a parallel strategy that combines pipeline parallelism and priority scheduling.

[0127] Considering that when the training resource requirement of any modality is less than or equal to the first threshold, its memory usage may be close to the hardware limit, or its computation time may be significantly different from other modalities. If the same parallel strategy is used in both cases, it may lead to memory overflow or serious device idleness, affecting the stability and efficiency of training. Therefore, based on the initial determination to adopt a data parallel strategy, this embodiment further judges based on sub-indicators such as memory utilization and modal heterogeneity, so as to set different parallel strategies for the two cases of high memory usage and unbalanced computational performance.

[0128] Specifically, if the training resource requirement of any modality is less than or equal to the first threshold and the memory utilization is greater than the second threshold, it indicates that although the model can be deployed, the storage resources are already very tight. When performing gradient synchronization in data parallelism, an additional buffer is needed to store gradient copies, which can easily lead to memory overflow errors.

[0129] To address this issue, this embodiment employs a parallel strategy combining data parallelism and gradient compression. By combining data parallelism with gradient compression, the high throughput advantage of data parallelism is retained, while gradient compression significantly reduces the network bandwidth and temporary GPU memory usage required for gradient synchronization operations. This effectively mitigates the risks associated with high GPU memory utilization while ensuring stable model training. Gradient compression refers to reducing the data volume of the gradient tensor using a specific algorithm before performing cross-node gradient synchronization.

[0130] Furthermore, modal heterogeneity is an indicator used to measure the degree of imbalance in computation time among different modalities. When modal heterogeneity is too high, if all modalities use standard data parallelism, the computational resources of faster modalities will remain idle for a long time after completing their own computation, waiting for the slowest modalities to complete their computation, resulting in idle computational resources and a decrease in overall training throughput.

[0131] Therefore, in this embodiment, when the training resource requirement for any modality is less than or equal to a first threshold and the modality heterogeneity is greater than a preset third threshold, a strategy combining pipeline parallelism and priority scheduling is adopted. This treats different modalities as different stages of the pipeline, dividing the training data into multiple micro-batches. This allows a fast modality to immediately begin processing the next micro-batch after processing the current one, without waiting for a slow modality, thus achieving computational overlap and reducing device idle time. Simultaneously, the priority scheduler identifies slow modalities that act as performance bottlenecks and assigns them higher execution priority, ensuring that computations on the critical path are not preempted, thereby maximizing the overall efficiency of the pipeline.

[0132] Through the refined strategy selection rules described above, this embodiment considers not only the resource requirements of the modality itself when determining the parallel strategy, but also the memory pressure that the modality may encounter in actual operation and the performance imbalance with other modalities. This makes the selection of parallel strategies more targeted and adaptable, and can effectively deal with the memory bottleneck in data parallelism and the time difference in multimodal training, thereby achieving efficient utilization of computing resources in more complex scenarios.

[0133] For example, assuming a multimodal model includes text, image, and audio modalities, this embodiment can determine the following differentiated parallel strategy combinations for them:

[0134] For text-based modalities, modules such as self-attention mechanisms are typically computationally intensive. To alleviate the excessive computational burden on individual computing nodes and address the memory bottleneck caused by large model parameters, a parallel strategy combining model parallelism and tensor parallelism can be employed. For example, in a parallel group consisting of 8 GPUs, the query (Q), key (K), and value (V) matrices in the self-attention mechanism can be partitioned along the attention head dimension, with each GPU only responsible for computing a portion of the head results. This distributes the computational load and parameter storage pressure evenly across the 8 GPUs.

[0135] For image modalities, the number of parameters is usually large. If conventional data parallelism is used, gradient synchronization across nodes will generate huge communication overhead, exhibiting a high sensitivity to communication bandwidth. Therefore, a parallel strategy combining data parallelism and gradient compression techniques can be adopted. For example, before each gradient synchronization, a Top-K gradient sparsity algorithm can be used, selecting only the top 30% of the largest gradient values ​​for communication, thereby reducing the amount of data for a single gradient synchronization by about 70%, significantly reducing the network burden.

[0136] For audio modalities, the processing flow often exhibits inherent stage-based sequential dependencies. Pipeline parallelism is the most effective strategy to address this characteristic. For example, the entire processing flow can be divided into multiple stages and placed on different GPUs to form a pipeline. Simultaneously, priority scheduling can be combined to set computational stages on the critical path as high-priority tasks and allocate them independent computational flows, ensuring their priority execution and preventing them from being blocked by other computational tasks, thereby maximizing the overall efficiency of the pipeline.

[0137] Based on any of the above embodiments, the modal characteristic indices of each mode in the multimodal model are obtained, including:

[0138] Dynamic code analysis is performed on the multimodal model to statistically analyze the computational characteristics of each modality, including the number of floating-point operations per second, the number of parameters, and the proportion of computationally intensive layers.

[0139] The training data of the multimodal model is pre-scanned, and the data characteristic indicators of each modality are statistically analyzed. The data characteristic indicators include single sample data size, batch size range and data loading time.

[0140] Perform simulation training of the multimodal model and record the communication characteristic indicators of each modality. The communication characteristic indicators include the communication synchronization frequency, cross-modal interaction data volume, and communication impact degree of each training step.

[0141] Specifically, dynamic code analysis can be understood as the process of real-time monitoring and data collection of the computational behavior of each component of the model (i.e., each modal encoder, fusion layer, etc.) in a simulated or actual operating environment, through tools or code instrumentation.

[0142] Floating-point operations per second (FLOPs) refers to the total number of floating-point operations performed by the model to complete one forward or backward propagation of a corresponding modality. It is a core indicator for measuring computational complexity; a higher FLOP count indicates a more computationally intensive task and a greater demand for computing resources. The number of parameters refers to the total number of trainable parameters in a corresponding modality or the amount of storage space they occupy, reflecting the model's storage requirements and some computational overhead. The proportion of computationally intensive layers refers to the percentage of floating-point operations performed by the most computationally intensive key layers in a modality relative to the total computational load of that modality. It reveals the distribution characteristics of the modality's computational load; a high proportion means that optimizing these key layers can lead to significant performance improvements.

[0143] The training data for the multimodal model undergoes sample pre-scanning to statistically obtain data characteristic indicators for each modality. Sample pre-scanning refers to a rapid and representative sampling and analysis of the training dataset before formal training begins to obtain the physical and processing characteristics of the data. Data characteristic indicators include single-sample data size, batch size range, and data loading time. Single-sample data size refers to the storage space occupied by a single data sample after loading into memory and undergoing preprocessing; it characterizes the basic memory bandwidth requirements of the data. Batch size range refers to the effective range of the number of samples that can be processed in a single training iteration, under the premise of meeting hardware constraints and algorithm convergence requirements; it characterizes the trade-off between training throughput and hardware resources. Data loading time refers to the total time required to read a batch of data from the storage medium, complete all preprocessing steps, and send it to the computing unit; it characterizes the efficiency of the data input pipeline.

[0144] Communication characteristics of each modality can be obtained by performing simulated training of the multimodal model. These characteristics include the communication synchronization frequency at each training step, the amount of cross-modal interaction data, and the degree of communication impact. Here, simulated training typically refers to running multiple complete training iterations on a target distributed cluster using small-scale data. Its purpose is not model convergence, but rather to capture and record cross-node communication behavior. The communication synchronization frequency at each training step refers to the frequency of communication between computing nodes for gradient aggregation during distributed training, usually related to the frequency of training steps, and characterizes the tightness of coupling between computation and communication. The amount of cross-modal interaction data refers to the total amount of intermediate feature data or parameters that needs to be transmitted between computing nodes of different modalities to achieve intermodal information fusion, and characterizes the specific network bandwidth requirements of the modality fusion stage. The degree of communication impact refers to the extent to which increased network latency negatively affects training time or overall throughput, characterizing the modality's dependence on network quality. High sensitivity indicates the need for parallel strategies that can hide communication latency.

[0145] Based on any of the above embodiments, and based on the parallel strategy for each modality, the training process of a multimodal model is executed in a distributed computing environment, including:

[0146] The parallel strategies for each modality are integrated into the training execution engine, and the training process of the multimodal model is executed in a distributed computing environment according to the parallel strategies for each modality.

[0147] Specifically, the training execution engine is used to transform parallel strategies into a sequence of computation and communication instructions that can be executed on distributed hardware, and to uniformly schedule, monitor and manage the entire training process.

[0148] After integrating the parallel strategies for each modality into the training execution engine, the training process of the multimodal model is executed in a distributed computing environment according to the parallel strategies for each modality. This training process can be driven by the task scheduling and execution module within the training execution engine.

[0149] As a preferred embodiment, the training execution engine can have built-in monitoring functions to collect resource utilization data (such as GPU utilization and video memory usage) of each computing node in real time. When unbalanced or inefficient resource utilization is detected, the engine can trigger the aforementioned dynamic adjustment mechanism, re-call steps 110-130 to generate a new parallel strategy, and seamlessly switch to the new execution scheme, thereby achieving continuous optimization throughout the training process.

[0150] This embodiment integrates and executes differentiated parallel strategies by introducing a training execution engine. It encapsulates and automates complex and error-prone hybrid parallel logic. Compared to solutions that require application developers to manually write large amounts of low-level code to handle interactions between different parallel modes, this embodiment significantly reduces the difficulty of developing and deploying distributed training of complex multimodal models, thereby enhancing the system's robustness and scalability.

[0151] Based on any of the above embodiments, the process of training a multimodal model in a distributed computing environment further includes:

[0152] By deploying a monitoring agent in a distributed computing environment, resource usage information for each modality is collected in real time.

[0153] When the training duration reaches the preset duration or when the resource usage information collected in real time triggers the preset conditions, the process returns to the steps of obtaining modal characteristic indicators, calculating training resource requirements, determining parallel strategies, and training models.

[0154] Specifically, a monitoring agent can be understood as a lightweight software process or service deployed on each computing node in a distributed computing environment. The monitoring agent is responsible for collecting data on the underlying hardware resource usage, such as the real-time utilization of the GPU, memory usage, and network throughput.

[0155] Resource usage information refers to quantitative data collected by the monitoring agent that reflects the actual hardware resources consumed by each modality during training. For example, resource usage information may include an average GPU utilization rate of 85% and a peak memory usage of 30GB for the GPU group allocated to the image modality; and an average GPU utilization rate of only 30% and a peak memory usage of 10GB for the GPU group allocated to the text modality.

[0156] Considering that the training process of a multimodal model is a long-term dynamic process, the characteristics of the model and its resource requirements will change in different training stages. In order to ensure that the parallel strategy can always adapt to the current training state and maintain the highest resource utilization efficiency, this embodiment returns to the steps of obtaining modal characteristic indicators, calculating training resource requirements, determining parallel strategies and training models when the training time reaches the preset time or the resource occupancy information collected in real time triggers the preset conditions.

[0157] When the training duration reaches the preset duration, it indicates that the model may have entered a new training phase. If the initial parallel strategy is still used at this point, there may be a mismatch between the strategy and the current training characteristics, leading to wasted resources or performance degradation. Therefore, after the training duration reaches the preset duration, the process returns to the steps of obtaining modality characteristic indicators, calculating training resource requirements, determining the parallel strategy, and training the model. The preset duration can be set to N training iteration cycles (e.g., 1000 steps) or a fixed time interval T (e.g., 30 minutes).

[0158] Furthermore, when real-time resource usage information triggers preset conditions, it indicates that the current system is experiencing actual resource imbalance or performance bottlenecks. If adjustments are not made promptly, continuous resource waste may occur, potentially even leading to training interruptions. Therefore, after real-time resource usage information triggers preset conditions, the system returns to the steps of acquiring modality characteristic indicators, calculating training resource requirements, determining parallel strategies, and training models. These preset conditions can include low utilization conditions (e.g., the average GPU utilization for a certain modality consistently below 30%), high load bottleneck conditions (e.g., the GPU memory utilization for a certain modality consistently above 95%), or abnormal execution time proportion conditions (e.g., the computation time for a certain modality exceeds 70% of the total iteration time).

[0159] When any of the above conditions are met, the process will return to the steps of obtaining modal characteristic metrics, calculating training resource requirements, determining the parallel strategy, and training the model. In other words, the training execution engine will pause the current training task and re-execute steps 110, 120, and 130, thereby generating a new, differentiated parallel strategy better suited to the current state. The engine will then apply this new strategy to the training task and continue executing step 140.

[0160] Based on any of the above embodiments Figure 2 This is a flowchart illustrating another distributed training method for multimodal models provided by the present invention, as shown below. Figure 2 As shown, the method includes:

[0161] First, the modal characteristic indicators of each modality in the multimodal model are obtained through the modal characteristic extraction module. Specifically, the training data, model structure, and hardware environment information of the multimodal model are input into the modal characteristic extraction module. The module then performs quantitative analysis on the characteristics of different modalities to construct a characteristic indicator library. For example, computational characteristic indicators, including the number of floating-point operations per second and the number of parameters, are obtained through dynamic code analysis; data characteristic indicators, including the size of a single sample and the range of batch sizes, are obtained through sample pre-scanning; and communication characteristic indicators, including the communication synchronization frequency and the degree of communication impact for each training step, are obtained through simulated training records.

[0162] Then, the hybrid parallel strategy generation module calculates the training resource requirements for each modality based on data from the feature index library, and determines a differentiated parallel strategy for each modality accordingly. For example, for the text modality, which is "computationally intensive and has low communication sensitivity," a strategy combining "tensor parallelism and model parallelism" is adopted to reduce computational pressure; for the image modality, which has "large data volume and moderate communication sensitivity," a strategy combining "data parallelism and gradient compression" may be adopted to reduce network transmission overhead.

[0163] Next, the resource dynamic allocation module allocates computing nodes corresponding to each mode according to the parallel strategy. The training execution engine integrates the parallel strategy on each computing node and coordinates with each computing node to complete the parameter update of the model.

[0164] In addition, during the training process, the real-time monitoring module collects resource usage information for each modality in real time through monitoring agents deployed on each computing node, and inputs the collected information into the dynamic resource allocation module. The dynamic resource allocation module makes decisions based on the real-time load, dynamically adjusts resource allocation and parallel strategies, and sends adjustment instructions to the training execution engine, thereby achieving dynamic optimization of training resources.

[0165] Finally, during the training process, the model training results are fed back to the modal feature extraction module. The modal feature extraction module can then re-evaluate the feature metrics based on the model training results that reflect the new training state information. In turn, the hybrid parallel policy generation module can update the parallel policy, ensuring high efficiency throughout the entire training lifecycle.

[0166] The distributed training device for multimodal models provided by the present invention will be described below. The distributed training device for multimodal models described below can be referred to in correspondence with the distributed training method for multimodal models described above.

[0167] Based on any of the above embodiments Figure 3 This is a schematic diagram of the structure of the distributed training device for the multimodal model provided by the present invention, as shown below. Figure 3 As shown, the device includes:

[0168] The acquisition unit 310 is used to acquire the modal characteristic indices of each mode in the multimodal model. The modal characteristic indices of each mode are used to characterize the computational characteristics, data characteristics and communication characteristics of each mode.

[0169] The computing unit 320 is used to calculate the training resource requirements of each modality based on the modal characteristic indicators of each modality.

[0170] The determination unit 330 is used to determine the parallel strategy differently for each modality based on the training resource requirements of each modality;

[0171] Training unit 340 is used to perform the training process of multimodal models in a distributed computing environment based on parallel strategies for each modality.

[0172] Figure 4 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 4 As shown, the electronic device may include a processor 410, a communications interface 420, a memory 430, and a communication bus 440. The processor 410, communications interface 420, and memory 430 communicate with each other via the communication bus 440. The processor 410 can call logical instructions from the memory 430 to execute a distributed training method for a multimodal model.

[0173] Furthermore, the logical instructions in the aforementioned memory 430 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to related technologies, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0174] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the distributed training method for the multimodal model provided by the above methods.

[0175] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a distributed training method for the multimodal model provided by the methods described above.

[0176] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0177] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of software products. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0178] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A distributed training method for a multimodal model, characterized in that, include: The modal characteristic indices of each modality in the multimodal model are obtained. These indices are used to characterize the computational, data, and communication characteristics of each modality. The multimodal model refers to a deep learning model that integrates multiple data modalities for joint training. Calculate the training resource requirements for each modality based on its modal characteristic indices; Based on the training resource requirements of each modality, a differentiated parallel strategy is determined for each modality; the differentiated determination of the parallel strategy means selecting the most suitable combination of parallel techniques for each modality or combination of modalities according to the training resource requirements of different modalities. The training process of the multimodal model is executed in a distributed computing environment based on the parallel strategy of each modality. The modal characteristic indicators for each mode include computational characteristic indicators, data characteristic indicators, and communication characteristic indicators; The step of calculating the training resource requirements for each modality based on its modal characteristic indices includes: Based on the computational characteristic indices of each mode, determine the parameter quantities of each mode; Based on the data characteristic indicators of each modality, determine the ratio between the data volume of other modalities and the data volume of each modality; Based on the communication characteristic indicators of each mode, determine the maximum bandwidth and current bandwidth of the environment in which each mode is located; The training resource requirements for each modality are calculated based on the number of parameters for each modality, the ratio between the amount of data in other modalities and the amount of data in each modality, and the difference between the maximum bandwidth and the current bandwidth of the environment in which each modality is located. The acquisition of modal characteristic indices for each mode in the multimodal model includes: Dynamic code analysis is performed on the multimodal model to statistically analyze the computational characteristic indicators of each modality, including the number of floating-point operations per second, the number of parameters, and the proportion of computationally intensive layers. The training data of the multimodal model is pre-scanned, and the data characteristic indicators of each modality are statistically analyzed. The data characteristic indicators include single sample data size, batch size range, and data loading time. The simulation training of the multimodal model is performed, and the communication characteristic indicators of each modality are recorded. The communication characteristic indicators include the communication synchronization frequency, the amount of cross-modal interaction data, and the degree of communication impact for each training step.

2. The distributed training method for multimodal models according to claim 1, characterized in that, The training resource requirements for each modality are calculated based on the parameter quantity of each modality, the ratio between the data quantity of other modalities and the data quantity of each modality, and the difference between the maximum bandwidth and the current bandwidth of the environment in which each modality is located. This includes: The training resource requirements for each modality are obtained by weighting and summing the logarithms of the parameters of each modality, the ratio between the data volume of other modalities and the data volume of each modality, and the difference between the maximum bandwidth of the environment in which each modality is located and the current bandwidth.

3. The distributed training method for a multimodal model according to any one of claims 1 to 2, characterized in that, The method for determining parallel strategies differentiated for each modality based on its training resource requirements includes: If the training resource requirement of any modality exceeds the first threshold, then the corresponding modality is determined to adopt a parallel strategy that combines tensor parallelism and model parallelism. If the training resource requirement of any modality is less than or equal to the first threshold, then the corresponding modality is determined to adopt a data parallel strategy.

4. The distributed training method for multimodal models according to claim 3, characterized in that, If the training resource requirement of any modality is less than or equal to the first threshold, then determining that the corresponding modality adopts a data-parallel strategy includes: If the training resource requirement of any modality is less than or equal to the first threshold and the memory utilization is greater than the second threshold, then the corresponding modality is determined to adopt a parallel strategy that combines data parallelism and gradient compression. If the training resource requirement of any modality is less than or equal to the first threshold and the modality heterogeneity is greater than the third threshold, then the corresponding modality is determined to adopt a parallel strategy that combines pipeline parallelism and priority scheduling.

5. The distributed training method for a multimodal model according to any one of claims 1 to 2, characterized in that, The parallel strategy based on each modality, which executes the training process of the multimodal model in a distributed computing environment, includes: The parallel strategies for each modality are integrated into the training execution engine, and the training process of the multimodal model is executed in a distributed computing environment according to the parallel strategies for each modality.

6. The distributed training method for a multimodal model according to any one of claims 1 to 2, characterized in that, The training process of the multimodal model in a distributed computing environment also includes: By deploying a monitoring agent in the distributed computing environment, resource usage information for each modality is collected in real time. When the training duration reaches the preset duration or when the resource usage information collected in real time triggers the preset conditions, the process returns to the steps of obtaining modal characteristic indicators, calculating training resource requirements, determining parallel strategies, and training models.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the distributed training method for the multimodal model as described in any one of claims 1 to 6.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the distributed training method for the multimodal model as described in any one of claims 1 to 6.

Images