Distributed multi-modal training reasoning method and system for heterogeneous nodes and medium

Abstract

The invention relates to the technical field of distributed computing, and discloses a distributed multi-modal training reasoning method and system for heterogeneous nodes and a medium, and the method comprises the steps: analyzing a distributed deployment topological structure of a multi-modal model, obtaining a connection relation between modal branches, and generating a modal interaction dependency graph; applying the unified precision configuration of the precision cooperation group to an input end of a cross-modal interaction layer, configuring a precision alignment operator, and outputting interactive input data after precision alignment; obtaining the hardware precision capability of each heterogeneous node in the reasoning stage, and generating a training-reasoning precision migration mapping table; and based on the training-reasoning precision migration mapping table, generating precision configuration and a distributed reasoning scheduling scheme in a reasoning stage. The technical effects of guaranteeing the stability of the model training numerical value and improving the convergence quality are achieved.

Description

Technical Field

[0001] This invention relates to the field of distributed computing technology, and more specifically, to a distributed multimodal training and inference method, system, and medium for heterogeneous nodes. Background Technology

[0002] In distributed training and inference scenarios for ultra-large-scale multimodal models, the multimodal model consists of multiple branches that process data from different modalities, such as vision branches, language branches, and audio branches. These branches need to perform feature interaction and fusion at cross-modal attention layers. To improve computational efficiency, heterogeneous node clusters are used in actual deployments to perform distributed computing. Different nodes are equipped with different types of processors and accelerators, and the supported numerical precision formats vary.

[0003] Existing technologies employ an independent hybrid precision configuration strategy, where each modal branch selects a precision format such as FP32, FP16, or BF16 based on its own computational characteristics. During inference deployment, the precision configuration scheme determined in the training phase is directly transferred to the inference nodes. In distributed training, a multi-stage pipeline execution is used, including intra-modal pipelines and inter-modal pipelines.

[0004] However, existing technologies have the following technical problems: First, when modal branches with different precision configurations perform feature fusion in the cross-modal interaction layer, the differences in numerical range and representation precision can lead to abnormal numerical calculations, manifested as gradient explosion, underflow, or precision loss, affecting the model convergence quality. Second, the precision configuration scheme determined during the training phase often does not match the hardware precision capabilities of the heterogeneous nodes used for inference deployment when transferred to the inference phase, leading to inference failure or additional overhead from full precision conversion. Third, bubble periods exist in the execution of multi-stage pipelines. If two-stage pipelines are scheduled independently, bubble periods may overlap, causing computing resources to be completely idle and wasted during those periods. Summary of the Invention

[0005] This invention provides a distributed multimodal training and inference method, system, and medium for heterogeneous nodes, solving the technical problems of abnormal multimodal interaction layer values, training-inference accuracy transfer mismatch, and resource waste during pipeline bubble periods in related technologies.

[0006] This invention provides a distributed multimodal training and inference method for heterogeneous nodes, comprising: Analyze the distributed deployment topology of the multimodal model, obtain the connection relationships between each modal branch, and generate a modal interaction dependency graph; Based on the modal interaction dependency graph, modal branches with direct interaction connections are grouped into the same precision coordination group. Combining the hardware precision capability parameters of each heterogeneous node, a unified precision configuration is determined for each coordination group under the condition of satisfying precision compatibility constraints, and a coordination precision allocation scheme is generated. The unified precision configuration of the precision coordination group is applied to the input end of the cross-modal interaction layer. A precision alignment operator is configured to perform scaling and format conversion operations on feature data from different modal branches and output precision-aligned interactive input data. Distributed training of the multimodal model is performed according to the collaborative precision allocation scheme. Each heterogeneous node performs calculations according to the precision configuration of its respective collaborative group. Feature fusion is performed in the cross-modal interaction layer through the precision alignment operator, and the model parameters are iteratively updated until convergence. The hardware precision capabilities of each heterogeneous node in the inference phase are obtained, and the collaborative precision allocation scheme in the training phase is mapped and analyzed with the precision capabilities of the inference nodes to generate a training-inference precision transfer mapping table. Based on the training-inference precision transfer mapping table, weight format conversion and scaling factor recalculation are performed on modal branches that require precision adjustment to generate precision configuration and distributed inference scheduling scheme for the inference phase.

[0007] Furthermore, the precision alignment operator uses a dynamic quantization method to calculate the scaling factor. For the input feature tensor, the scaling factor is calculated by dividing the maximum absolute value of all elements in the input feature tensor by the maximum positive value that can be represented by the target precision format. The scaled tensor is obtained by dividing the original tensor by the scaling factor, and the original numerical range is restored by multiplying by the scaling factor after interactive calculation.

[0008] Furthermore, the step of grouping modal branches with direct interactive connections into the same precision coordination group includes: extracting all modal branch pairs with edge connections from the modal interaction dependency graph, merging modal branches with transitive dependencies into the same coordination group using a disjoint-set data structure algorithm; for each coordination group, collecting the hardware precision capability parameters of the nodes deployed by each modal branch within the group, taking the intersection of the precision supported by each node as the optional precision set of the group, and selecting the precision format with the best computational efficiency and meeting the model precision requirements from the optional precision set as the unified precision configuration of the coordination group.

[0009] Furthermore, when performing distributed training, it also includes: performing joint scheduling analysis on the multi-level pipeline topology, identifying the bubble position, bubble duration, and available computing resources for each stage of the intra-modal pipeline and each stage of the inter-modal pipeline, and generating a bubble distribution map and a bubble resource list.

[0010] Furthermore, it also includes: extracting the precision format conversion operations to be performed from the precision switching plan, estimating the computational cost of each conversion operation, matching the conversion tasks with bubble resources, prioritizing the allocation of conversion tasks to bubble time slots that can be accommodated by the duration and resources, and generating a bubble filling scheduling scheme; wherein, the computational cost of the conversion operation is obtained by multiplying the number of elements of the tensor to be converted by the number of conversion operations for each element, and the execution time is obtained by dividing the total computational cost by the computing power of the available computing resources.

[0011] Furthermore, for conversion tasks whose computational load exceeds the capacity of a single bubble, they are split into multiple subtasks and executed in multiple consecutive bubble time periods. At the same time, hierarchical buffers are configured on each level of pipeline node to allocate independent buffer space for intramodal data flow and intermodal data flow.

[0012] Furthermore, the hierarchical buffer is configured with a two-level structure at each pipeline node. The first level is an intra-modal buffer that stores the intermediate results of the current modal pipeline, and the second level is an inter-modal buffer that stores the input and output data of cross-modal interactions. The address spaces of the two levels of buffers are independent.

[0013] Furthermore, the computation start timing of each heterogeneous node is coordinated according to the bubble filling scheduling scheme. When the pipeline execution enters the bubble period, the corresponding precision conversion task is triggered. The converted data is written to the spare area of ​​the double buffer. When the next stage of computation needs the data, it is read from the spare area and the buffer role is switched.

[0014] Furthermore, the dual buffer includes two storage spaces: a primary region and a backup region. During the current computation phase, input data is read from the primary region, while the precision conversion task writes the results to the backup region. When the conversion task is completed and the next computation phase is about to begin, the roles of the primary region and the backup region are reversed, and the original backup region becomes the new primary region for the next phase to read.

[0015] This invention provides a distributed multimodal training and inference system with heterogeneous nodes, comprising: The topology analysis module is used to analyze the distributed deployment topology of the multimodal model, obtain the connection relationships between each modal branch, and generate a modal interaction dependency graph. The precision coordination allocation module is used to group modal branches with direct interactive connections into the same precision coordination group based on the modal interaction dependency graph, and determine a unified precision configuration for each coordination group by combining the hardware precision capability parameters of each heterogeneous node. The precision alignment module is used to configure precision alignment operators at the input of the cross-modal interaction layer and perform scaling and format conversion operations on feature data from different modal branches; The distributed training module is used to perform distributed training of multimodal models according to a collaborative precision allocation scheme. The precision transfer mapping module is used to perform mapping analysis between the collaborative precision allocation scheme in the training phase and the precision capability of the inference nodes, and generate a training-inference precision transfer mapping table. The inference adaptation module is used to perform weight format conversion and scaling factor recalculation based on the training-inference precision transfer map, and generate precision configuration and distributed inference scheduling scheme for the inference stage.

[0016] The beneficial effects of this invention are as follows: This invention establishes a modal interaction dependency graph and groups modal branches with direct interactions into the same precision coordination group based on this graph. It assigns a uniform precision configuration to all branches within the coordination group, thus solving the numerical anomaly problem caused by the mismatch of precision formats in multimodal interaction layers. It overcomes the technical defects of gradient explosion, underflow, or precision loss in cross-modal computation under the independent precision allocation strategy, and achieves the technical effect of ensuring the numerical stability of model training and improving convergence quality.

[0017] This invention performs dynamic scaling and format conversion on input features at the entry point of the cross-modal interaction layer by configuring a precision alignment operator, which normalizes feature data from different numerical ranges to a unified interval, further ensuring the numerical stability of cross-modal fusion computation.

[0018] This invention solves the problem of inference failure or accuracy loss caused by direct migration of training accuracy by establishing a training-inference accuracy transfer mapping table, analyzing the matching relationship between training accuracy configuration and inference node hardware capabilities before inference deployment, and performing accuracy format conversion and scaling factor recalculation on mismatched layers. This enables the accuracy coordination relationship determined in the training phase to adapt to the actual capabilities of heterogeneous nodes in the inference phase, achieving the technical effects of improving the success rate of inference deployment and reducing accuracy conversion overhead.

[0019] This invention identifies bubble periods and their available resources through joint scheduling analysis of multi-stage pipelines, and injects precision format conversion tasks into these bubble periods for execution. This solves the technical problem of wasted computing resources during completely idle bubble periods, making use of previously idle computing resources. Simultaneously, it avoids conversion operations occupying critical path execution time, achieving the technical effects of improving computing resource utilization and shortening total training time. The hierarchical buffer and double buffer configuration ensures that the writing of precision conversion data and the reading of normal computation data do not interfere with each other, guaranteeing the continuity of pipeline execution. Attached Figure Description

[0020] Figure 1 This is a flowchart of the distributed multimodal training and inference method for heterogeneous nodes according to the present invention; Figure 2 This is a line graph showing the changes in performance indicators during the training process of this invention; Figure 3 This is a bar chart comparing the accuracy and throughput of heterogeneous nodes according to the present invention; Figure 4 This is a bar chart comparing the accuracy format scoring of this invention; Figure 5 This is a bar chart showing the inference performance of the present invention under different batch sizes. Detailed Implementation

[0021] The subject matter described herein will now be discussed with reference to exemplary embodiments. It should be understood that these embodiments are discussed only to enable those skilled in the art to better understand and implement the subject matter described herein, and changes may be made to the function and arrangement of the elements discussed without departing from the scope of this specification. Various processes or components may be omitted, substituted, or added as needed in the examples. Furthermore, some features described in the examples may be combined in other examples.

[0022] At least one embodiment of the present invention discloses a distributed multimodal training and inference method for heterogeneous nodes, such as Figure 1 As shown, it includes the following steps: Step 1: Analyze the distributed deployment topology of the ultra-large-scale multimodal model, obtain the pipeline stages within each modality, the parallelism between modalities, and the connection relationships between branches of each modality, and generate a multi-level pipeline topology and modal interaction dependency graph.

[0023] Specifically, the network structure of the multimodal model is traversed to identify the hierarchical divisions and pipeline split points of each modal encoder branch, and the data flow within each modality is recorded. Simultaneously, all cross-modal attention layers and feature fusion layers are identified, and the input sources and output destinations of these interaction layers are extracted to establish dependencies between modal branches. The modal interaction dependency graph uses modal branches as nodes and cross-modal interaction connections as edges, with edge attributes including the location identifier of the interaction layer and data dimension information.

[0024] Furthermore, the identification of pipeline split points employs a computational load balancing strategy. The rationale for this strategy is that by evenly distributing the total computational load across each pipeline stage, the execution times of each stage can be made as similar as possible, thereby minimizing pipeline bubble periods and improving overall throughput. Specifically: For each layer in the modal encoder branch, calculate its parameter count and computational complexity according to the preset pipeline stages. The total computational cost of the branches Average score There are several stages, and the target computational cost for each stage is... Starting from the model input layer, accumulate the computational cost of each layer. When the accumulated sum reaches or exceeds... At this point, a breakpoint is set after the current pipeline stage to mark the end of the current stage and the beginning of the next stage; this process continues until all stages are completed. Setting the split points.

[0025] The computational complexity of a layer is calculated based on the number of multiply-accumulate operations performed in that layer. For a convolutional layer, the computational complexity is O(n). The computational complexity for a fully connected layer is O(n). The computational complexity of the attention layer is O(n). The relevant parameters are obtained directly from the model structure definition. Number of pipeline stages. The method for determining this is based on the number of available heterogeneous nodes. and model branch depth ,set up ,in This refers to the minimum number of layers contained in a single pipeline stage, typically taking the value of [value missing]. This ensures that each pipeline stage has sufficient computational resources to amortize communication overhead.

[0026] Furthermore, the parameters in the computational complexity formula are defined as follows: For convolutional layers, This refers to the batch size of the output feature maps for this layer. Input the number of channels. Number of output channels Let be the side length of the convolution kernel. and These are the height and width of the output feature map, respectively. These parameters are directly read from the model structure configuration file or network definition code. For example, when the convolutional layer is configured as "Number of input channels = 64, Number of output channels = 128, Kernel size = 3×3, Input feature map size = 56×56, Batch size = 32",... but , , , , For the attention layer, Given the length of the input sequence, For example, when processing a text sequence of length 512 and the embedding dimension of each word is 768, , .

[0027] Figure 3 The computational throughput of eight heterogeneous nodes in the training cluster was compared under different precision formats (FP32, FP16, BF16).

[0028] Step 2: Based on the modal interaction dependency graph, modal branches with direct interaction connections are grouped into the same precision coordination group. Combining the hardware precision capability parameters of each heterogeneous node, the precision configuration is determined for each coordination group under the condition of satisfying precision compatibility constraints, and a coordination precision allocation scheme and precision switching plan are generated.

[0029] Specifically, all modal branch pairs with edge connections are extracted from the modal interaction dependency graph. A disjoint-set data structure (DFS) algorithm is used to merge modal branches with transitive dependencies into the same cooperative group. For each cooperative group, the hardware precision capability parameters of the nodes deployed in each modal branch are collected, and the intersection of the precisions supported by each node is taken as the optional precision set for that group. The precision format with the best computational efficiency and that meets the model precision requirements is selected from the optional precision set as the unified precision configuration for that cooperative group. The precision switching plan records the transition position and direction from input precision to cooperative group precision at each layer.

[0030] Furthermore, the input to the precision collaboration group establishment method is the modal interaction dependency graph and the hardware precision capability parameters of each heterogeneous node, and the output is the collaborative precision allocation scheme and precision switching plan. In this embodiment, the disjoint-set data structure algorithm is used to merge transitive dependencies. Specifically, during initialization, each modal branch is treated as an independent set. Each edge in the modal interaction dependency graph is traversed, and for each connecting modal branch... and modal branching The edge, of which and For modal branch indices, a merge operation is performed to unite the sets containing two branches into one set. After traversing all edges, modal branches belonging to the same set constitute a precision co-group. The precision format is selected using a weighted scoring method, applying a weighted scoring method to each precision format in the available precision set. Calculate its score ,in For precision format Average computational throughput across all nodes in the collaborative group For precision format The numerical representation of precision. and The highest-scoring precision format is selected as the uniform precision configuration for the collaborative group, serving as the weighting coefficient.

[0031] Furthermore, weighting coefficients and satisfy and The constraints, among which Reflecting the importance of computational efficiency, The weighting ratios reflect the importance of numerical accuracy and are set according to the specific application scenario. The specific method for determining the weighting coefficients is as follows: For training tasks, to ensure numerical stability, set... and For inference tasks, accuracy is prioritized; however, for computational efficiency, the settings are adjusted accordingly. and Prioritize throughput; for specific application scenarios, adjust the weighting ratios within the above range based on the relative requirements for latency and accuracy. When multiple accuracy formats exist for the scoring... When they are equal, the numerical representation precision should be selected first. Higher-precision formats are selected to ensure the numerical stability of model training; if the numerical representation precision is also the same, the format ranked higher in the optional precision set is selected, with the sorting rule being the priority order of FP32, BF16, and FP16.

[0032] It should be noted that hardware precision capability parameters refer to the list of numerical precision formats supported by each heterogeneous node and their corresponding computational throughput. For example, a GPU node supports FP32, FP16, and BF16 formats, with FP16 having the highest tensor core throughput. To eliminate the difference in dimensions between computational throughput and numerical representation precision, a computational scoring system is used... Previously, respectively for and Perform normalization and scale it to... The normalization process uses the min-max normalization method. The rationale for choosing the min-max normalization method is that it preserves the relative magnitudes of the original data and ensures that the normalized values ​​are strictly limited to a certain range. Within the interval, weighted summation is convenient for calculation. Specifically, for throughput calculation, minimum-maximum normalization is used. ,in and These represent the minimum and maximum throughput values ​​for each precision format in the optional precision set; for numerical representation precision, the number of significant digits is mapped to... interval, ,in For precision format The number of significant digits, and These are the minimum and maximum number of significant digits in the optional precision set, respectively.

[0033] Figure 4 Compare the performance of FP32 and FP16 precision formats in three dimensions: throughput score, numerical precision score, and overall score.

[0034] Step 3: Apply the unified precision configuration of the precision coordination group to the input of the cross-modal interaction layer, configure the precision alignment operator, perform scaling and format conversion operations on the feature data from different modal branches, eliminate the inconsistency of numerical range caused by the difference in original precision, and output the precision-aligned interactive input data.

[0035] Specifically, the precision alignment operator is inserted at the forward computation entry point of the cross-modal interaction layer. It receives feature tensors from each modal branch, first detects the current precision format of each input tensor, and performs precision conversion on tensors that are inconsistent with the target precision of the cooperating group. Then, it calculates the numerical statistics of each input tensor, including the maximum absolute value and dynamic range, and calculates the scaling factor based on the statistics to scale each input tensor to a uniform numerical range. Finally, it outputs the aligned tensors to the cross-modal attention computation or feature fusion operation.

[0036] Furthermore, the precision alignment operator takes as input the feature tensors from each modal branch and the target precision format of the co-op group, and outputs the precision-aligned feature tensors. The precision alignment operator performs operations including format conversion and normalization in data preprocessing. Format conversion unifies numerical data from different precision formats to the target precision format, while normalization dynamically scales tensors with different numerical ranges to a uniform interval, ensuring numerical stability in subsequent cross-modal interactive computations.

[0037] In this embodiment, to reduce information loss during precision alignment, the precision alignment operator employs a dynamic quantization method to calculate the scaling factor. For the input feature tensor... scaling factor The calculation formula is:

[0038] in, Tensor The maximum absolute value of all elements in the set. This represents the maximum positive value that the target precision format can represent. The scaled tensor. pass Obtained by inverse scaling after interactive calculation is complete. Restore the original value range.

[0039] Furthermore, The value is determined according to the target precision format, specifically: For FP32 format For FP16 format, For BF16 format, Scaled tensor The input is fed into the cross-modal interaction layer for feature fusion computation, and the output tensor of the interaction layer is... For a result tensor that has already been scaled, an inverse scaling operation is required. The output tensor is restored to its original numerical range. This is so that subsequent network layers can continue processing.

[0040] Furthermore, when That is, set the scaling factor when all elements of the tensor are zero. At this point, both scaling and inverse scaling operations are identity transformations, keeping the zero tensor unchanged and directly outputting it to subsequent calculations.

[0041] Step 4: Perform distributed training of the multimodal model according to the collaborative precision allocation scheme and precision alignment configuration. Each heterogeneous node performs modal encoding calculation according to the precision configuration of its respective collaborative group. Feature fusion is performed through the precision alignment operator in the cross-modal interaction layer. The model parameters are iteratively updated until convergence. The trained model weights and collaborative precision configuration records are output.

[0042] Furthermore, the criterion for model convergence is: continuity The decrease in the validation set loss function within a training cycle is less than a preset threshold. Or, training reaches the preset maximum number of iterations, where The number of consecutive periods used for convergence determination. The preset threshold is set according to the specific model size and training task; a typical value is [value to be filled in]. and .

[0043] Furthermore, the model parameter updates employ an adaptive learning rate optimization algorithm, specifically the AdamW optimizer, which combines the adaptive learning rate adjustment and weight decay regularization of the Adam algorithm.

[0044] Initial learning rate The typical value range is set according to the model size and batch size. The learning rate is dynamically adjusted during training using a cosine annealing strategy. The learning rate for each training step is , in Pi To minimize the learning rate, Total training steps. Minimum learning rate. The method for determining this is: setting This is one percent of the initial learning rate, ensuring that the learning rate does not drop too low and cause training to stagnate; total training steps Based on the size of the training dataset Batch size and the preset number of training cycles calculate, ,in The floor sign is the number of training cycles. Based on the model size and task complexity, a typical value for a large-scale multimodal model is [value to be filled in]. to The momentum parameter of the AdamW optimizer is set to... and The weight decay coefficient is set to .

[0045] Figure 2 This demonstrates the changing trends of validation set loss and gradient norm of a multimodal model during distributed training.

[0046] Step 5: Obtain the hardware precision capability description and deployment topology information of each heterogeneous node in the inference phase, perform mapping analysis on the collaborative precision allocation scheme generated in the training phase and the precision capability of the inference nodes, identify modal branches and interaction layers that do not match the training precision configuration and the inference node capability, and generate a training-inference precision transfer mapping table.

[0047] Specifically, the precision capability parameters of each inference node and the connection topology between nodes are read from the inference cluster configuration and compared item by item with the collaborative precision allocation scheme in the training phase. For each precision collaboration group, it is checked whether the precision format determined in the training phase is in the support list of the corresponding inference node; for cases where there is a mismatch, the precision collaboration group, the modal branches involved, the original precision format, and the optional precision format of the target node are recorded in the migration mapping table.

[0048] Furthermore, the training-inference precision transfer mapping method takes as input the collaborative precision allocation scheme during the training phase, the hardware precision capability parameters of each inference node, and the inference deployment topology information, and outputs a training-inference precision transfer mapping table. This table includes the identifiers of the collaborative groups requiring precision adjustment, a list of involved modal branches, the original precision format used during training, and the target precision format selectable by the inference nodes, providing guidance for subsequent precision conversion and weight adaptation.

[0049] Step 6: Based on the training-inference precision transfer mapping table, perform weight format conversion and scaling factor recalculation on the modal branches and interaction layers that require precision adjustment, rebuild the precision co-group relationship in the distributed deployment of inference nodes, and generate the precision configuration and distributed inference scheduling scheme for the inference phase.

[0050] Specifically, for each layer marked in the transfer map table that needs adjustment, the target precision is selected from the optional precisions of the inference nodes, with the format closest to the original training precision being chosen. Precision format conversion is then performed on the weight parameters of that layer. Simultaneously, the scaling factor corresponding to that layer is recalculated to adapt to the new precision range. During inference deployment, the collaborative group relationships between each modality branch are reconstructed according to the new precision configuration, generating a node task allocation and execution order scheduling scheme for the inference phase.

[0051] Furthermore, the precision adjustment and co-group reconstruction methods take as input a training-inference precision transfer map, trained model weights, and hardware precision capability parameters of the inference nodes, and output model weights adapted to the inference nodes, precision configuration for the inference phase, and a distributed inference scheduling scheme. Precision format conversion employs a proximity matching principle, selecting the format closest to the training precision numerical representation range and precision from the inference node's available precision set to minimize precision loss introduced by the conversion. The scaling factor is recalculated based on the numerical range of the new target precision format, ensuring that cross-modal interactive computation during the inference phase maintains numerical stability consistent with the training phase.

[0052] Furthermore, the distance metric between precision formats uses the absolute value of the difference in numerical representation capability, specifically: for training precision formats... and inference node optional precision format Calculate the difference in the number of significant digits. ,in and The number of significant digits for training accuracy and inference accuracy are selected respectively. The format with the lowest inference precision is chosen as the closest format. When multiple precision formats exist... When values ​​are equal, the format with the higher throughput is preferred.

[0053] Step 7: When an inference request arrives, the multimodal input data is distributed to the corresponding heterogeneous nodes according to the distributed inference scheduling scheme. Each node performs modal encoding and cross-modal fusion calculation according to the inference accuracy configuration, and the processing results of each node are aggregated and the final inference result is output.

[0054] Building upon step 4, the following steps are also included to improve training efficiency: Step 401: Perform joint scheduling analysis on the multi-stage pipeline topology, identify the bubble position, bubble duration, and available computing resources for each stage of the intra-modal pipeline and each stage of the inter-modal pipeline, and generate a bubble distribution map and a bubble resource list.

[0055] Specifically, the execution timing of a multi-stage pipeline is simulated, recording the start and end times of computation for each pipeline stage, and identifying waiting gaps between adjacent stages as bubble periods. For each bubble period, its duration is calculated, and the available computing units of the corresponding node at that time are queried as available resources. A bubble distribution diagram records the position of each bubble on the timeline, and a bubble resource list records the duration and available computing power of each bubble.

[0056] Furthermore, the bubble analysis method takes as input a multi-stage pipeline topology, estimated computation time for each pipeline stage, and hardware resource configuration for each node, and outputs a bubble distribution map and a bubble resource list. Bubble identification employs a time-series simulation method, advancing the time according to the pipeline execution sequence. When a stage cannot start due to waiting for upstream data, this waiting period is designated as a bubble period. Bubble resource statistics are based on node-level hardware resource monitoring, querying the computing units and memory resources not occupied by any pipeline stage during the bubble period.

[0057] Furthermore, the pipeline execution timing simulation adopts a discrete event simulation method, specifically: initializing the time axis. ,in To simulate time, an event queue is created for each pipeline stage. Each event includes an event type and a trigger time. According to pipeline dependencies, when a preceding stage completes data preparation, the start event for that stage is inserted into the event queue. The event trigger time is the completion time of the preceding stage plus the data transmission delay. The time-series simulation processes all events in the event queue in chronological order, recording the start and end times of each stage. The end time is the start time plus the computation time of that stage. When two adjacent stages are executed on the same node and the end time of the preceding stage is earlier than the start time of the subsequent stage, the time interval between the two times is the bubble period. The computation time of each pipeline stage is estimated using performance analysis methods. For each stage, a forward propagation is performed on the target node, and the actual execution time is recorded as the estimated computation time for that stage. For stages not actually deployed, theoretical estimations are made based on the computational complexity of that stage and the computing power parameters of the target node. The estimation formula is... ,in The computational complexity of the stage. Node computing power. Node computing power parameters. The methods for obtaining this power are: reading the theoretical peak computing power from the node's hardware specifications, or measuring the actual computing power by running standard benchmark programs. For GPU nodes, the computing power parameter is measured in TFLOPS (trillion floating-point operations per second), and a typical GPU computing power range is... to TFLOPS.

[0058] Step 402: Extract the precision format conversion operations to be performed from the precision switching plan, estimate the computational cost of each conversion operation, match the conversion tasks with bubble resources, prioritize the allocation of conversion tasks to bubble time slots that can be accommodated by the duration and resources, and generate a bubble filling scheduling scheme.

[0059] Specifically, for each conversion operation in the precision switching plan, the required computation and execution time are estimated based on the tensor size and conversion type involved. The conversion operations are sorted from smallest to largest according to their execution time requirements, and bubble time slots that meet the duration requirements are searched in the bubble resource list for each conversion operation. Successfully matched conversion operations are marked as executed within the bubble, while unsuccessfully matched operations are retained for execution on the critical path.

[0060] Furthermore, the bubble filling scheduling method takes as input a precision switching plan, a bubble resource list, and computational cost estimates for each conversion operation, and outputs a bubble filling scheduling scheme. The matching of conversion tasks and bubbles employs an optimal adaptation strategy. For each conversion operation, the bubble with the most available resources is selected from the bubble time slots that meet the duration constraints for allocation, thereby improving resource utilization. The matching process uses a greedy algorithm, allocating tasks sequentially according to their execution time from shortest to longest, prioritizing small-scale conversion operations in fragmented, short bubble time slots, and allocating large-scale conversion operations to continuous, long bubble time slots.

[0061] Furthermore, the bubble filling scheduling problem can be formalized as an optimization problem with a time dimension, and the objective function is:

[0062] in, For the index of the transformation operation, This represents the total number of precision conversion operations. For decision variables, when the transformation operation When assigned to execute during bubble time ,otherwise ; For conversion operation The execution time is considered. The optimization objective function represents minimizing the total time of transformation operations performed on the critical path, i.e., maximizing the proportion of transformation operation time performed during bubble periods. Constraints of the optimization problem include: Duration constraint: ,in For the index of bubble periods, Bubble period Duration; Resource constraints: ,in For conversion operation Number of computing units required Bubble period Number of available computing units; Mutual exclusion constraint: At most one conversion operation is allocated per bubble period. ,in Indicates conversion operation Allocated to bubble time slot Unique allocation constraint: Each conversion operation can be assigned to at most one bubble time slot. ,in This represents the total number of bubble periods.

[0063] Furthermore, the specific definition of the duration constraint is: the execution time of the conversion operation. It must be less than or equal to the duration of the bubble phase. ,Right now Only bubble periods that meet this constraint are included in the candidate set. The sufficiency of available resources is measured using the resource utilization rate index for bubble periods. Number of available computing units and conversion operations Number of computing units required Resource sufficiency is defined as ,choose The bubble with the highest value is allocated to ensure that the conversion operation receives sufficient computing resources.

[0064] It should be noted that the computational complexity estimation method for the conversion operation is as follows: Let the number of elements in the tensor to be converted be... The number of transformation operations for each element is (Determined by the source accuracy and target accuracy), the total computational cost of this transformation operation is... , in The total computational cost of the transformation operation and the execution time. ,in This refers to the computing power of available computing resources.

[0065] Furthermore, the transformation operands for each element The precision format is determined based on the source and target precision formats. Specifically, for operations converting from FP32 to FP16 or BF16, This includes one range check and one truncation / rounding operation; for conversion from FP16 or BF16 to FP32, Only one zero-expansion operation is required for the conversion between FP16 and BF16. It needs to be converted to FP32 first, and then to the target format. The specific implementation of the conversion operation calls the precision conversion instructions provided by the hardware. On GPUs that support mixed-precision computing, the execution latency of the precision conversion instructions is 1-3 clock cycles.

[0066] Step 403: For large-scale conversion tasks whose computational load exceeds the capacity of a single bubble, split them into multiple subtasks and distribute them across multiple consecutive bubble time periods. At the same time, configure hierarchical buffers on each level of pipeline node to allocate independent buffer space for intra-modal data streams and inter-modal data streams, and generate a sharded conversion plan and buffer configuration.

[0067] Specifically, for transformation tasks that cannot be completed in a single bubble, the ratio of the total execution time required for large-scale transformation tasks to the available bubble duration is calculated. Based on this ratio, the tensor to be transformed is split into multiple sub-tensors along the batch or feature dimension. The transformation of each sub-tensor is assigned as an independent subtask to consecutive bubble time slots. A hierarchical buffer is configured with a two-level structure at each pipeline node: the first level is an intra-modal buffer storing intermediate results for the current modality, and the second level is an inter-modal buffer storing input and output data for cross-modal interactions. The address spaces of the two buffer levels are independent to avoid contention.

[0068] Step 404: Coordinate the computation start timing of each heterogeneous node according to the bubble filling scheduling scheme. When the pipeline execution enters the bubble period, the corresponding precision conversion task is triggered. The converted data is written to the spare area of ​​the double buffer. When the next stage of computation needs the data, it is read from the spare area and the buffer role is switched to ensure the normal progress of the pipeline within the modality and the parallel execution of precision conversion. The optimized training iteration results are output.

[0069] Figure 5 Compare inference latency and inference throughput performance under different batch sizes.

[0070] In this embodiment, the dual-buffer working mode is as follows: the buffer contains two storage spaces, a primary area and a backup area. During the current computation phase, input data is read from the primary area, while the precision conversion task writes the result to the backup area. When the conversion task is completed and the next computation phase is about to begin, the roles of the primary and backup areas are reversed, with the original backup area becoming the new primary area for the next phase to read. Through this mode, the precision conversion write operation and the normal computation read operation access different storage areas, eliminating data access conflicts.

[0071] This implementation identifies the interaction relationships between modal branches in a multimodal model by establishing a modal interaction dependency graph. Based on this dependency graph, modal branches with direct interactions are grouped into the same precision coordination group. Under the premise of satisfying the hardware capability constraints of each heterogeneous node, a uniform precision configuration is assigned to all branches within the coordination group. Since the modal branches within the same coordination group use the same numerical precision format, no numerical range differences caused by precision format mismatches occur during feature fusion across modal interaction layers, thus overcoming the numerical anomalies that occur in cross-modal computation under independent precision allocation strategies.

[0072] This implementation uses a precision alignment operator to perform dynamic scaling and format conversion on the input features at the entry point of the cross-modal interaction layer, normalizing feature data from different numerical ranges to a unified interval, thereby further ensuring the numerical stability of cross-modal fusion computation.

[0073] This implementation establishes a training-inference accuracy transfer mapping table, analyzes the matching relationship between training accuracy configuration and inference node hardware capabilities before inference deployment, and performs accuracy format conversion and scaling factor recalculation on mismatched layers. This enables the accuracy coordination relationship determined in the training phase to adapt to the actual capabilities of heterogeneous nodes in the inference phase, overcoming inference failure or accuracy loss caused by direct transfer of training accuracy.

[0074] This implementation identifies bubble periods and their available resources through joint scheduling analysis of the multi-stage pipeline, injecting precision format conversion tasks into these bubble periods for execution. This utilizes previously idle computing resources while avoiding the conversion operation consuming critical path execution time. The configuration of layered buffers and dual buffers ensures that the data writing for precision conversion and the data reading for normal computation do not interfere with each other, guaranteeing the continuity of pipeline execution.

[0075] The key differences between this implementation and existing technologies include: First, a scheme that establishes precision collaboration groups based on modal interaction dependency graphs and implements unified precision configuration within these groups. Existing hybrid precision training methods typically configure precision independently according to network layer type or single-modal characteristics, without considering precision compatibility constraints in multimodal interaction scenarios. Second, a scheme that matches and schedules precision format conversion tasks with pipeline bubble periods. Existing pipeline scheduling methods mainly focus on load balancing of computational tasks, without considering auxiliary computations such as precision conversion in the utilization of bubble periods. Third, a training-inference precision transfer mapping mechanism. Existing solutions typically employ globally unified quantization or precision conversion strategies during model deployment, without fine-tuning the collaboration group structure of multimodal models and the differentiated capabilities of heterogeneous inference nodes. These key differences address the technical problems of precision collaboration, resource utilization, and cross-stage transfer in multimodal distributed training and inference, demonstrating significant substantive features and substantial progress.

[0076] A video content platform needs to deploy an intelligent video analytics system to automatically understand multimodal information such as visual scenes, speech dialogues, and background music in videos, enabling content annotation, sentiment analysis, and intelligent recommendations. The system employs a trimodal fusion model comprising a visual encoder, a speech encoder, and an audio encoder, with a model parameter scale of 12 billion. Training is required on a distributed cluster consisting of eight heterogeneous GPU nodes, and inference services are deployed on four inference acceleration nodes. The training cluster includes four NVIDIA A100 nodes and four NVIDIA V100 nodes, while the inference cluster consists of two NVIDIA T4 nodes and two Huawei Ascend 910 nodes, with significant differences in hardware precision capabilities among the nodes.

[0077] Step 1 Implementation: A topological analysis is performed on the trimodal fusion model. The visual encoder contains a 32-layer hybrid structure of convolutional and Transformer layers, the language encoder contains a 24-layer Transformer structure, and the audio encoder contains a 16-layer one-dimensional convolutional and recurrent neural network structure. Based on a computational balance strategy, the visual encoder is divided into 4 pipeline stages, each containing 8 network layers; the language encoder is divided into 3 pipeline stages, each containing 8 network layers; and the audio encoder is divided into 2 pipeline stages, each containing 8 network layers. Cross-modal interaction layers in the recognition model, including the visual-language interaction layer at layer 16, the visual-audio interaction layer at layer 20, and the trimodal fusion layer at layer 24, are used to establish a modal interaction dependency graph containing 3 modal branch nodes and 3 interaction edges.

[0078] Step 2 Implementation: Based on the modal interaction dependency graph, a disjoint-set data structure algorithm is used to establish precision coordination groups. Since there are transitive dependencies between the visual, language, and audio branches, the three modal branches are grouped into the same precision coordination group. Hardware precision capability parameters for each node are collected; all nodes support FP32 and FP16 precision, with the A100 node additionally supporting BF16 precision. For the selectable precision set... Perform scoring calculations and set weighting coefficients. and The FP16 format achieves a tensor core throughput of 312 TFLOPS on the A100 node and 125 TFLOPS on the V100 node, with an average throughput of 218.5 TFLOPS. The FP32 format achieves an average throughput of 19.6 TFLOPS. After normalization, the computational throughput score for FP16 is... The numerical accuracy score is Overall score FP32's overall score The highest-scoring FP32 is selected as the unified precision configuration for the collaborative group, generating a collaborative precision allocation scheme. The precision switching plan records that precision alignment operations need to be performed at the inputs of the three cross-modal interaction layers (Layer-16, Layer-20, Layer-24) to ensure that the features of each modal branch are uniformly converted to FP32 format before entering the interaction layer.

[0079] Step 3 implementation: Configure precision alignment operators at the inputs of the three cross-modal interaction layers.

[0080] At the vision-language interaction layer, feature tensors from the vision branch are received. and the feature tensor of language branches Calculate the maximum absolute value of each tensor. , According to the target accuracy FP32 Calculate scaling factor and The scaling operation is performed on the two tensors, and the output precision-aligned feature tensor is used by the cross-modal attention layer. Similarly, corresponding precision alignment operators are configured at the inputs of the visual-audio interaction layer and the trimodal fusion layer to complete the precision alignment configuration of all cross-modal interaction layers.

[0081] Step 4: Perform distributed training on the 8 heterogeneous nodes of the training cluster, configured with FP32 precision. Use the AdamW optimizer and set the initial learning rate. Minimum learning rate Number of training cycles Batch size The training dataset contains 5 million video clips, with a total training step count of [number missing]. Step. After 80 training epochs, the validation set loss decreased from the initial 2.847 to 0.124, with the loss decreasing by less than [amount missing] for five consecutive epochs. Once the convergence condition is met, output the training completed model weights and collaborative accuracy configuration records.

[0082] Step 5 Implementation: Obtain the hardware precision capability parameters of the inference cluster. Two T4 nodes support FP32, FP16, and INT8 precision, and two Ascend 910 nodes support FP32 and FP16 precision, but not BF16 precision. Map the FP32 precision configuration determined during the training phase to the inference node capabilities. Check the FP32 configuration of precision coordination group Group-1; it is found that all inference nodes support FP32 precision, and the precision format is a perfect match. Generate a training-inference precision migration mapping table, recording that coordination group Group-1 uses FP32 precision on all four inference nodes without requiring format conversion. The mapping table marks the matching status of all nodes as "Yes," and the target inference precision is FP32.

[0083] Step 6 Implementation: Based on the training-inference precision transfer mapping table, since the training precision and inference precision are perfectly matched, there is no need to perform weight format conversion and scaling factor recalculation. According to the co-group relationship in the training phase, the precision co-configuration of the three-modal branches is reconstructed in the inference deployment, confirming that co-group Group-1 continues to use FP32 precision in the inference phase. The first two pipeline stages of the visual encoder are deployed to two T4 nodes, and the last two stages are deployed to two Ascend 910 nodes; the pipeline stages of the speech encoder and audio encoder are allocated to four inference nodes according to the load balancing principle, generating the precision configuration for the inference phase (all modal branches and interaction layers use FP32 precision) and a distributed inference scheduling scheme (including node task allocation and execution order).

[0084] Step 7 Implementation: When a user uploads a video for content analysis, the system decomposes the video into three modal inputs: frame sequence, audio waveform, and subtitle text. Following a distributed inference scheduling scheme, each modal data is distributed to its corresponding node. Each node performs modal encoding according to its FP32 precision configuration. At the cross-modal interaction layer, features from each modality are aggregated and precision-aligned fusion calculations are performed. The final output includes analysis results containing scene category, sentiment, and key events. The inference latency for a single video segment is 180 milliseconds.

[0085] Heterogeneous node hardware precision capability parameters:

[0086] Modal interaction dependencies:

[0087] Precision Coordination Group Allocation Results:

[0088] Accuracy configuration and performance data during training phase:

[0089] Training-inference accuracy transfer mapping:

[0090] The embodiments of the present invention have been described above. However, the embodiments are not limited to the specific implementation methods described above. The specific implementation methods described above are merely illustrative and not restrictive. Those skilled in the art can make more equivalent embodiments under the guidance of the present embodiments, and all of them are within the protection scope of the present embodiments.

Claims

1. A distributed multimodal training and inference method for heterogeneous nodes, characterized in that, Includes the following steps: Analyze the distributed deployment topology of the multimodal model, obtain the connection relationships between each modal branch, and generate a modal interaction dependency graph; Based on the modal interaction dependency graph, modal branches with direct interaction connections are grouped into the same precision coordination group. Combining the hardware precision capability parameters of each heterogeneous node, a unified precision configuration is determined for each coordination group under the condition of satisfying precision compatibility constraints, and a coordination precision allocation scheme is generated. The unified precision configuration of the precision coordination group is applied to the input end of the cross-modal interaction layer. A precision alignment operator is configured to perform scaling and format conversion operations on feature data from different modal branches and output precision-aligned interactive input data. Distributed training of the multimodal model is performed according to the collaborative precision allocation scheme. Each heterogeneous node performs calculations according to the precision configuration of its respective collaborative group. Feature fusion is performed in the cross-modal interaction layer through the precision alignment operator, and the model parameters are iteratively updated until convergence. The hardware precision capabilities of each heterogeneous node in the inference phase are obtained, and the collaborative precision allocation scheme in the training phase is mapped and analyzed with the precision capabilities of the inference nodes to generate a training-inference precision transfer mapping table. Based on the training-inference precision transfer mapping table, weight format conversion and scaling factor recalculation are performed on modal branches that require precision adjustment to generate precision configuration and distributed inference scheduling scheme for the inference phase.

2. The distributed multimodal training and inference method for heterogeneous nodes according to claim 1, characterized in that, The precision alignment operator uses a dynamic quantization method to calculate the scaling factor. For the input feature tensor, the scaling factor is calculated by dividing the maximum absolute value of all elements in the input feature tensor by the maximum positive value that can be represented by the target precision format. The scaled tensor is obtained by dividing the original tensor by the scaling factor, and the original numerical range is restored by multiplying by the scaling factor after interactive computation is completed.

3. The distributed multimodal training and inference method for heterogeneous nodes according to claim 1, characterized in that, The step of grouping modal branches with direct interactive connections into the same precision coordination group includes: extracting all modal branch pairs with edge connections from the modal interaction dependency graph, merging modal branches with transitive dependencies into the same coordination group using a disjoint-set data structure algorithm; for each coordination group, collecting the hardware precision capability parameters of the nodes deployed by each modal branch within the group, taking the intersection of the precision supported by each node as the optional precision set of the group, and selecting the precision format with the best computational efficiency and that meets the model precision requirements from the optional precision set as the unified precision configuration of the coordination group.

4. The distributed multimodal training and inference method for heterogeneous nodes according to claim 1, characterized in that, The distributed training process also includes: joint scheduling analysis of the multi-stage pipeline topology, identifying the bubble position, bubble duration, and available computing resources for each stage of the intra-modal pipeline and each stage of the inter-modal pipeline, and generating a bubble distribution map and a bubble resource list.

5. The distributed multimodal training and inference method for heterogeneous nodes according to claim 4, characterized in that, It also includes: extracting the precision format conversion operations to be performed from the precision switching plan, estimating the computational cost of each conversion operation, matching conversion tasks with bubble resources, prioritizing the allocation of conversion tasks to bubble time slots that can be accommodated by the duration and resources, and generating a bubble filling scheduling scheme; wherein, the computational cost of the conversion operation is obtained by multiplying the number of elements of the tensor to be converted by the number of conversion operations for each element, and the execution time is obtained by dividing the total computational cost by the computing power of the available computing resources.

6. The distributed multimodal training and inference method for heterogeneous nodes according to claim 5, characterized in that, For conversion tasks whose computational load exceeds the capacity of a single bubble, they are split into multiple subtasks and executed across multiple consecutive bubble time periods. At the same time, hierarchical buffers are configured on each pipeline node to allocate independent buffer space for intra-modal data flow and inter-modal data flow.

7. The distributed multimodal training and inference method for heterogeneous nodes according to claim 6, characterized in that, The hierarchical buffer is configured with a two-level structure at each pipeline node. The first level is an intra-modal buffer that stores the intermediate results of the current modality pipeline, and the second level is an inter-modal buffer that stores the input and output data of cross-modal interactions. The address spaces of the two levels of buffers are independent.

8. The method according to claim 5, characterized in that, The bubble filling scheduling scheme coordinates the computation start timing of each heterogeneous node. When the pipeline enters the bubble period, the corresponding precision conversion task is triggered. The converted data is written to the spare area of ​​the double buffer. When the next stage of computation needs the data, it is read from the spare area and the buffer role is switched.

9. The distributed multimodal training and inference method for heterogeneous nodes according to claim 8, characterized in that, The dual buffer contains two storage spaces: a main region and a backup region. During the current computation phase, input data is read from the main region, while the precision conversion task writes the results to the backup region. When the conversion task is completed and the next stage of calculation is about to begin, the roles of the primary and backup regions are swapped, and the original backup region becomes the new primary region for the next stage to read.

10. A distributed multimodal training and inference system for heterogeneous nodes, used to execute the distributed multimodal training and inference method for heterogeneous nodes as described in any one of claims 1-9, characterized in that, include: The topology analysis module is used to analyze the distributed deployment topology of the multimodal model, obtain the connection relationships between each modal branch, and generate a modal interaction dependency graph. The precision coordination allocation module is used to group modal branches with direct interactive connections into the same precision coordination group based on the modal interaction dependency graph, and determine a unified precision configuration for each coordination group by combining the hardware precision capability parameters of each heterogeneous node. The precision alignment module is used to configure precision alignment operators at the input of the cross-modal interaction layer and perform scaling and format conversion operations on feature data from different modal branches; The distributed training module is used to perform distributed training of multimodal models according to a collaborative precision allocation scheme. The precision transfer mapping module is used to perform mapping analysis between the collaborative precision allocation scheme in the training phase and the precision capability of the inference nodes, and generate a training-inference precision transfer mapping table. The inference adaptation module is used to perform weight format conversion and scaling factor recalculation based on the training-inference precision transfer map, and generate precision configuration and distributed inference scheduling scheme for the inference stage.

Images