Data scheduling method and device for multi-modal model training, equipment, medium and product

Abstract

The present disclosure provides a data scheduling method and device for multi-modal model training, equipment, medium and product, which can be applied to the field of large language model and artificial intelligence technology. The method comprises the following steps: writing a plurality of sample embedding representations obtained by encoding a plurality of multi-modal samples by an encoder into a storage pool, and writing addressing identifiers indicating the storage positions of the sample embedding representations in the storage pool into the sample pool. The data amount of the addressing identifier is smaller than that of the sample embedding representation; based on the addressing identifier, the sample embedding representation corresponding to the addressing identifier is read from the storage pool by using a scheduler; a plurality of data packets obtained by packing the sample embedding representation are sent to a decoder for processing to obtain a processing result, and a multi-modal model is trained based on the loss between the processing result and a true label.

Description

Technical Field

[0001] This disclosure relates to the fields of large language models and artificial intelligence, and more specifically, to a data scheduling method, apparatus, device, medium, and product for training multimodal models. Background Technology

[0002] Multimodal Large Language Model (MLLM) is widely used in scenarios such as autonomous driving, embodied intelligence, visual question answering, and e-commerce search recommendation. By combining the reasoning ability of large language models with the multimodal processing ability of multimodal models, alignment between visual features and language features is achieved.

[0003] In realizing the present invention, the inventors discovered that the training of multimodal models in related technologies has at least the following problems: module heterogeneity and data heterogeneity limit the resource utilization and adaptability of training multimodal large language models. Summary of the Invention

[0004] In view of this, this disclosure provides a data scheduling method, apparatus, device, medium, and product for multimodal model training.

[0005] One aspect of this disclosure provides a data scheduling method for training a multimodal model, the multimodal model including encoders and decoders deployed independently in different device groups, the method comprising: writing multiple sample embedding representations obtained by encoding multimodal samples using the encoder into a storage pool, and writing an addressing identifier indicating the storage location of each sample embedding representation in the storage pool into a sample pool, the data size of the addressing identifier being smaller than the data size of the sample embedding representation; based on the addressing identifier, using a scheduler to read the sample embedding representation corresponding to the addressing identifier from the storage pool; sending multiple data packets obtained by packaging the sample embedding representations to the decoder for processing to obtain a processing result, and training the multimodal model based on the loss between the processing result and the true label.

[0006] According to embodiments of this disclosure, multiple sample embedding representations obtained by encoding multimodal samples using an encoder are written into a storage pool, and addressing identifiers indicating the storage location of each sample embedding representation in the storage pool are written into the sample pool. This includes: when multiple sample embedding representations are encapsulated into tensor objects using a tensor management tool and addressing identifiers are assigned to each tensor object, the sample embedding representations are written into the physical address corresponding to the addressing identifier in the storage pool, and the addressing identifiers are written into the sample pool. The encoder continues to process subsequent multimodal samples after writing the sample embedding representations into the storage pool, without waiting for the decoder to complete the reading operation of the sample embedding representations.

[0007] According to embodiments of this disclosure, based on an addressing identifier, a scheduler reads a sample embedding representation corresponding to the addressing identifier from a storage pool, including: the scheduler, which contains the sample pool, reads the addressing identifier from the sample pool, the addressing identifier containing the virtual address of the sample embedding representation in the storage pool; the scheduler queries a global page table in the storage pool based on the virtual address to obtain the physical address of the sample embedding representation in the physical storage medium, the global page table indicating the mapping relationship between virtual addresses and physical addresses across nodes in each training process; the scheduler selects a transmission path based on the physical address of the sample embedding representation and the physical location of the current training process of the asynchronous scheduler; and the scheduler reads the sample embedding representation from the physical address through the transmission path to package the read sample embedding.

[0008] According to embodiments of this disclosure, the storage pool maintains a global page table composed of local page tables of each training process; the method further includes: allocating a plurality of consecutive virtual addresses to each training process; allocating physical addresses corresponding to the plurality of virtual addresses from the storage pool to obtain a mapping relationship between virtual addresses and physical addresses; synchronizing the mapping relationship of each training process to other processes, and updating the local page tables in other processes used to store the mapping relationship between virtual addresses and physical addresses of each process, so as to update the global page table based on the updated local page tables.

[0009] According to embodiments of this disclosure, the data scheduling method for multimodal model training further includes: pre-running the encoder and decoder respectively to obtain a first throughput of the encoder and a second throughput of the decoder; determining a first number of encoder cards and a second number of decoder cards based on the ratio of the first throughput to the second throughput, so that the amount of data produced by the encoder per unit time matches the amount of data consumed by the decoder; and independently deploying a first device group of the encoder and a second device group of the decoder based on the first number of encoder cards and the second number of decoder cards respectively.

[0010] According to embodiments of this disclosure, the data scheduling method for multimodal model training further includes: mapping the video memory or memory of each computing node with an encoder and a decoder deployed to a virtual address space; and using the virtual address in the virtual address space as an addressing identifier indicating the storage location of each sample embedding in the storage pool.

[0011] According to embodiments of this disclosure, the data scheduling method for multimodal model training further includes: the decoder initiating a storage access request for data packets in the storage pool through an addressing identifier; selecting a transmission path that matches the storage location and the physical location of the device group where the decoder is located, based on the storage location in the addressing identifier and the physical location of the device group where the decoder is located, and transmitting the data packets to the decoder.

[0012] According to embodiments of this disclosure, selecting a transmission path that matches the storage location and the physical location includes: when the storage location and the physical location are different devices in the same node, using an intra-node high-speed interconnect transmission path as the transmission path; and when the storage location and the physical location are across nodes, using a remote direct memory access path as the transmission path.

[0013] According to embodiments of this disclosure, the data scheduling method for multimodal model training further includes: using a scheduler to obtain metadata provided by the encoder and storing it in a sample pool, the metadata including an address identifier and a sequence length; when the number of metadata in the sample pool is detected to be greater than a number threshold, or when the processing status of the decoder indicates that there are currently no data packets to be processed, the metadata in the sample pool is packaged to obtain multiple data packets, and the multiple data packets are sent to the decoder for processing.

[0014] According to embodiments of this disclosure, metadata in a sample pool is packaged to obtain multiple data packets, including: constructing a segment tree containing a preset integer range based on the sequence length of each of the multiple sequences in the metadata; the segment tree is used to manage multiple candidate packaged sequences; each leaf node in the segment tree is a packaging processing node in the scheduler used to process the sequences in the metadata; when a leaf node matching the resource required for the length of each candidate packaged sequence is found in the segment tree, the candidate packaged sequence is packaged using the leaf node to obtain multiple data packets, which are then sent to the decoder for processing.

[0015] According to embodiments of this disclosure, the data scheduling method for multimodal model training further includes: determining the execution cost of each packet processing node for packaging candidate packet sequences, the execution cost including a first component and a second component, the first component being positively correlated with the total number of valid tokens in all sample embeddings within a micro-batch, and the second component being positively correlated with the upper limit of the sequence length of each sample embedding within the packet processing node; clustering packet processing nodes with similar execution costs into an array of the same cost level based on the execution cost of each packet processing node, so that the execution delay of packet processing nodes in the array of the same cost level remains consistent; arranging each data packet in descending order of execution cost in the array of the same cost level, and sequentially allocating each data packet to each computing node of the decoder for processing using an alternating reverse traversal method, so that the execution cost of each computing node is consistent.

[0016] According to embodiments of this disclosure, multiple data packets obtained from the packetized sample embedding representation are sent to the decoder for processing to obtain processing results. The process includes: using a packetization tool associated with the scheduler to read the corresponding sample embedding representation from the storage pool according to the addressing identifier, and performing packetization processing to generate multiple data packets; sending the multiple data packets to each computing node of the decoder for decoding processing to obtain the processing results of multimodal samples; wherein the write operation of the sample pool and the read operation of the packetization tool are executed asynchronously in parallel.

[0017] Another aspect of this disclosure provides a data scheduling apparatus for training a multimodal model, the multimodal model including encoders and decoders deployed independently in different device groups, the apparatus comprising: a writing module for writing multiple sample embedding representations obtained by encoding multimodal samples using the encoder into a storage pool, and writing an addressing identifier indicating the storage location of each sample embedding representation in the storage pool into a sample pool, the data size of the addressing identifier being smaller than the data size of the sample embedding representation; a reading module for reading the sample embedding representation corresponding to the addressing identifier from the storage pool using a scheduler; and a sending module for sending multiple data packets obtained by packaging the sample embedding representations to the decoder for processing, obtaining a processing result, and training the multimodal model based on the loss between the processing result and the true label.

[0018] Another aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein, when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to perform the method as described above.

[0019] Another aspect of this disclosure provides a computer-readable storage medium storing computer-executable instructions that, when executed, are used to implement the method described above.

[0020] Another aspect of this disclosure provides a computer program product including computer-executable instructions that, when executed, are used to implement the method described above.

[0021] According to embodiments of this disclosure, by writing the sample embedding representation generated by the encoder into a storage pool and storing a lightweight addressing identifier in the sample pool, the scheduler asynchronously reads the corresponding sample embedding representation from the storage pool based on the addressing identifier and packages it for sending to the decoder. This achieves decoupled execution of the encoder and decoder in time. The encoder's "write-and-return" mechanism eliminates the need to wait for the decoder to finish processing, and the decoder's "on-demand retrieval" mechanism eliminates the need to wait for the encoder to complete the entire batch, avoiding batch-level synchronization barriers between traditional heterogeneous modules. Since the encoder and decoder are deployed independently in different device groups, they can each adopt the optimal parallelism, avoiding parallelism mismatch caused by module heterogeneity. In addition, the scheduler performs inter-process communication based on the lightweight addressing identifier rather than the large tensor sample embedding representation, which significantly reduces the communication overhead between heterogeneous modules and improves the resource utilization and scalability of multimodal large language model training. Attached Figure Description

[0022] The above and other objects, features, and advantages of this disclosure will become clearer from the following description of embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0023] Figure 1 This illustration schematically depicts an application scenario of a data scheduling method for multimodal model training according to embodiments of the present disclosure.

[0024] Figure 2 A flowchart illustrating a data scheduling method for multimodal model training according to an embodiment of the present disclosure is shown schematically.

[0025] Figure 3A This schematically illustrates a diagram of module-separate deployment during multimodal model training according to an embodiment of the present disclosure;

[0026] Figure 3B This illustration schematically shows a global virtual address space distribution in a storage pool according to an embodiment of the present disclosure;

[0027] Figure 3C The illustration shows a schematic diagram of an asynchronous data stream between an encoder and a decoder based on a storage pool, according to an embodiment of the present disclosure.

[0028] Figure 4 A schematic diagram of the structure of a scheduler according to an embodiment of the present disclosure is shown.

[0029] Figure 5 A block diagram of a data scheduling apparatus for multimodal model training according to an embodiment of the present disclosure is shown schematically.

[0030] Figure 6 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Detailed Implementation

[0031] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0032] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0033] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0034] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0035] A multimodal large language model mainly consists of an encoder that converts raw input information such as images, audio, and text into a unified vector representation within the model (called embedding), and a decoder that performs joint understanding and generation of the unified vector representation (also known as a large language model decoder). The encoder is relatively small and is frozen in most training tasks, only performing forward inference and not participating in training; the decoder is much larger and is the main source of computational power consumption for the entire model.

[0036] The inventors discovered two mismatches in the training of the two modules, leading to low model training efficiency. First, the models are heterogeneous: the encoder is small (typically no more than 1 byte of parameters), while the other is large (billions to hundreds of billions of parameters), resulting in differences in computational cost, memory usage, and optimal parallelization methods. Second, the data is heterogeneous: the training samples consist of a mixture of multimodal data (e.g., text, images, and audio), with significant differences in length and a long-tailed distribution; for example, text may be only tens of bytes long, while images can range from hundreds to thousands of bytes.

[0037] Related technologies struggle to simultaneously eliminate parallelism mismatch caused by heterogeneous modules in the model, uneven load caused by heterogeneous data, and batch-level synchronization barriers between heterogeneous modules, resulting in low computational utilization and scalability for training multimodal large language models.

[0038] In one example, the entire multimodal large language model is treated as a single network through monolithic training. A uniform tensor parallel / pipeline parallel (TP / PP) strategy is applied to both types of submodules, which is a typical single-program multiple data (SPMD) deployment. The encoder and decoder are placed in the same set of parallel devices and use the same degree of parallelism.

[0039] However, the optimal parallel strategies for the two types of submodules differ, resulting in overall inefficiency when forced to unify the degree of parallelism. The encoder, being small in size, can operate efficiently under pure data parallelism; the decoder, being larger in size, requires tensor parallelism / pipeline parallelism to meet its memory and computational needs. Imposing a uniform degree of parallelism on both during monolithic training makes it difficult to simultaneously match their optimal strategies, leading to an overall model computational efficiency (MFU) far lower than that achievable by each submodule individually.

[0040] In one example, the problem of model heterogeneity is mitigated by deploying different sub-modules to independent groups of devices and using their own customized parallel configurations through separate training.

[0041] However, in split training, the large differences in sample lengths and their long-tailed distribution lead to uneven load distribution among parallel data replicas. Since gradients need to be updated synchronously, slow nodes with high computational demands can hold back fast nodes, resulting in idle resources. Furthermore, traditional load balancing methods often measure load based on sequence length, which is difficult to adapt to the actual computational load after streaming loading and bundling (e.g., concatenating multiple variable-length samples into a fixed-length sequence). This results in low adaptability of traditional sequence length-based load balancing methods to the computational load after streaming loading and bundling.

[0042] Meanwhile, batch-level synchronization barriers exist between heterogeneous modules (encoder and decoder), and large tensor transfers are difficult to overlap with computation. Even in traditional solutions with separate deployments, the encoder and decoder still wait for each other in "batches": the decoder can only begin computation after all copies of the encoder have processed the current batch, resulting in idle time when the delays on both sides are inconsistent. The large tensor image embedding output by the encoder can have single-batch communication volumes ranging from hundreds of MB to GB, and since it is serial and difficult to overlap with computation, it further reduces resource utilization.

[0043] In view of this, embodiments of the present disclosure provide a data scheduling method, apparatus, device, medium, and product for multimodal model training. The method includes: writing multiple sample embedding representations obtained by encoding multimodal samples using an encoder into a storage pool; and writing an addressing identifier indicating the storage location of each sample embedding representation in the storage pool into a sample pool, wherein the data size of the addressing identifier is less than the data size of the sample embedding representation; based on the addressing identifier, using a scheduler to read the sample embedding representation corresponding to the addressing identifier from the storage pool; and sending multiple data packets obtained by packaging the sample embedding representations to a decoder for processing to obtain a processing result, thereby training a multimodal model based on the loss between the processing result and the true label.

[0044] According to embodiments of this disclosure, by writing the sample embedding representation generated by the encoder into a storage pool and storing a lightweight addressing identifier in the sample pool, the scheduler asynchronously reads the corresponding sample embedding representation from the storage pool based on the addressing identifier and packages it for sending to the decoder. This achieves decoupled execution of the encoder and decoder in time. The encoder's "write-and-return" mechanism eliminates the need to wait for the decoder to finish processing, and the decoder's "on-demand retrieval" mechanism eliminates the need to wait for the encoder to complete the entire batch, avoiding batch-level synchronization barriers between traditional heterogeneous modules. Since the encoder and decoder are deployed independently in different device groups, they can each adopt the optimal parallelism, avoiding parallelism mismatch caused by module heterogeneity. In addition, the scheduler performs inter-process communication based on the lightweight addressing identifier rather than the large tensor sample embedding representation, which significantly reduces the communication overhead between heterogeneous modules and improves the resource utilization and scalability of multimodal large language model training.

[0045] Figure 1 The diagram illustrates an application scenario of a data scheduling method for multimodal model training according to an embodiment of the present disclosure.

[0046] like Figure 1 As shown, the application scenario 100 according to this embodiment may include an encoder 101, a storage pool 102, a scheduler 103, and a decoder 104.

[0047] Encoder 101 can be the first processing module in a multimodal model, used to extract features from multimodal samples. Its input is raw multimodal data (such as images, text, audio, video, etc.), and its output is sample embedding representations. For example, encoder 101 encodes multimodal samples to obtain multiple sample embedding representations, providing basic data units for subsequent storage and scheduling.

[0048] For example, encoder 101 receives multimodal samples (e.g., 10 images and 10 corresponding text segments), obtains sample embedding representations through forward computation (e.g., a 768-dimensional vector for each sample), and writes these sample embedding representations into storage pool 102, assigning a unique storage address (e.g., addr_001, addr_002, etc.) to each embedding representation. Storage pool 102 can be memory or disk storage in a data scheduling system, used to store the sample embedding representations output by the encoder. Each sample embedding representation has a unique physical storage location (e.g., address offset or key value) in storage pool 102.

[0049] The scheduler 103 can be a control unit managing data flow, which can be collaboratively maintained by components such as a meta pool, a pack actor, a dispatch actor, and a global batch dictionary. For example, based on the real-time requirements of the training task (e.g., the load balancing strategy of the current batch), a set of addressing identifiers is selected from the meta pool, and a data read request is initiated from the storage pool 102 according to the selected addressing identifiers to obtain the corresponding sample embedding representation. The decoder 104 can be the second processing module in the multimodal model, used to receive data packets from the scheduler 103, perform forward inference on the data packets, and output the inference result or processing result.

[0050] It should be understood that Figure 1 The number of encoders, storage pools, schedulers, and decoders shown is merely illustrative. Depending on implementation requirements, any number of encoders, storage pools, schedulers, and decoders can be used.

[0051] Figure 2 A flowchart illustrating a data scheduling method for multimodal model training according to an embodiment of the present disclosure is shown schematically.

[0052] like Figure 2 As shown, the multimodal model includes encoders and decoders deployed independently in different device groups, and the method includes operations S201~S203.

[0053] In operation S201, multiple sample embedding representations obtained by encoding the multimodal samples using the encoder are written into the storage pool, and an addressing identifier indicating the storage location of each sample embedding representation in the storage pool is written into the sample pool. The data size of the addressing identifier is smaller than the data size of the sample embedding representation.

[0054] In the embodiments of this disclosure, the multimodal model is a multimodal large language model, such as a vision-language model (VLM). In the model training scenario, the encoder can be a processing module for feature extraction from multimodal samples, including but not limited to a vision encoder (ViT), a text segmenter, or an audio encoder. Multimodal samples can be training samples containing information from two or more modalities, such as text samples, image samples, video samples, and audio samples. The sample embedding representation can be a high-dimensional feature vector output by the encoder after performing forward computation on the multimodal samples.

[0055] The storage pool can be a centralized or distributed storage area, such as High Bandwidth Memory (HBM) and / or Dynamic Random Access Memory (DRAM) in a globally unified memory pool. Addressing identifiers can be lightweight metadata pointing to the storage location of sample embeddings within the storage pool, such as a Global Virtual Address (GVA), an OThandle, or a key-value index. The size of the addressing identifier data is smaller than the size of the sample embedding itself. The sample pool can be a collection of metadata (e.g., a queue, hash table, or dictionary) maintained by the scheduler, storing the addressing identifiers available for sample embeddings and their associated information (such as sequence length, modality type, etc.).

[0056] In operation S202, based on the addressing identifier, the scheduler reads the sample embedding representation corresponding to the addressing identifier from the storage pool.

[0057] In embodiments of this disclosure, the scheduler can be a scheduling module for data flow orchestration in a data scheduling system used for multimodal model training, and may include components such as a sample pool, triggering strategy, packaging execution body, and distribution execution body.

[0058] In operation S203, multiple data packets obtained from the embedded representation of the packaged samples are sent to the decoder for processing to obtain the processing result. A multimodal model is then trained based on the loss between the processing result and the true label.

[0059] In embodiments of this disclosure, the data packet can be a fixed-length sequence formed by packaging, containing a combination of multiple sample embeddings and necessary metadata. The decoder can be a Large Language Model Decoder (LLM Decoder) in a multimodal model, used to perform autoregressive or bidirectional inference on the packaged data packet, outputting the processing or inference result. The processing result can be a predicted value output by the decoder after forward inference on the data packet, such as a generated text sequence, classification label, or regression value. The ground truth label can be a pre-labeled correct result (ground truth) in the training data, serving as a supervision signal, and together with the processing result, calculates the loss to guide the parameter update of the multimodal model. The loss can characterize the difference between the processing result and the ground truth label, updating the model parameters through a backpropagation algorithm.

[0060] For example, before multimodal model training begins, the heterogeneous resource allocator pre-runs and measures the throughput of the encoder and decoder respectively, determines the number of cards and parallel configurations for each, and deploys resources and parallel strategies. During training, the encoder, acting as a producer, processes multimodal input, writes the encoded sample embedding representations into the storage pool and obtains the virtual address (GVA) of each sample embedding representation, and asynchronously sends lightweight metadata to the sample pool of the asynchronous scheduler without waiting for the decoder to finish processing. Based on the metadata stored in the sample pool, the scheduler triggers a balanced packing method, constructing balanced micro-batches or data packets. The decoder, acting as a consumer, retrieves the packed balanced micro-batches from the storage pool and performs forward and backward training. It can be understood that because the encoder and decoder are decoupled through the storage pool, they no longer wait for each other at batch boundaries, thus eliminating batch-level synchronization barriers.

[0061] For example, the visual encoder, acting as an encoder, is deployed on a set of independent GPUs. The storage pool is a globally unified memory pool, uniformly addressing the HBM and DRAM of all nodes using a 64-bit Global Virtual Address (GVA), also known as a virtual address. After each encoder process writes the sample embedding representation into its local memory, it makes it visible to all processes through page tables.

[0062] The sample pool in the scheduler stores the GVA, sequence length, modality type, etc., of each sample embedding representation. The scheduler can be a purely asynchronous scheduler, containing a triggering policy, a pack actor, a dispatch module, and a global batch dictionary. The decoder is deployed on a separate set of GPUs, using tensor parallelism (TP) and pipelined parallelism (PP).

[0063] For example, the encoder processes a batch of multimodal samples, writes the encoded sample embedding representations into a globally unified memory pool, and asynchronously sends the corresponding addressing identifiers to the scheduler's sample pool. The scheduler, based on a triggering strategy, selects a set of GVAs from the sample pool and reads the corresponding embedding representations from the memory pool via remote direct access or high-speed interoperability technology.

[0064] The packing execution unit within the scheduler packs the read sample embedding representations into fixed-length data packets, which are then evenly distributed to the computation nodes in each decoder. The decoder performs forward inference on the data packets, outputs the processing results, calculates the loss between the processed results and the real labels, and updates the parameters of the multimodal model based on the loss to obtain the updated multimodal model.

[0065] Figure 3A The diagram illustrates a modular deployment in multimodal model training according to an embodiment of the present disclosure.

[0066] like Figure 3A As shown, the encoder 101 and decoder 104 in multimodal model training can be decoupled into two independent execution entities through storage pool 102, and their deployment can be separated in terms of hardware resources (GPU), data flow, and scheduling logic. The encoder 101 is used for feature extraction of multimodal samples in multimodal model training, and the decoder 302 is used for text generation and inference in model training.

[0067] According to embodiments of this disclosure, by writing the sample embedding representation generated by the encoder into a storage pool and storing a lightweight addressing identifier in the sample pool, the scheduler asynchronously reads the corresponding sample embedding representation from the storage pool based on the addressing identifier and packages it for sending to the decoder. This achieves decoupled execution of the encoder and decoder in time. The encoder's "write-and-return" mechanism eliminates the need to wait for the decoder to finish processing, and the decoder's "on-demand retrieval" mechanism eliminates the need to wait for the encoder to complete the entire batch, avoiding batch-level synchronization barriers between traditional heterogeneous modules. Since the encoder and decoder are deployed independently in different device groups, they can each adopt the optimal parallelism, avoiding parallelism mismatch caused by module heterogeneity. In addition, the scheduler performs inter-process communication based on the lightweight addressing identifier rather than the large tensor sample embedding representation, which significantly reduces the communication overhead between heterogeneous modules and improves the resource utilization and scalability of multimodal large language model training.

[0068] As is understood, the above text has already explained the overall technical solution for how to schedule data during multimodal model training. The following text will explain how to determine the addressing identifier.

[0069] According to embodiments of this disclosure, the data scheduling method for multimodal model training further includes: mapping the video memory or memory of each computing node with an encoder and a decoder deployed to a virtual address space; and using the virtual address in the virtual address space as an addressing identifier indicating the storage location of each sample embedding in the storage pool.

[0070] In the embodiments of this disclosure, a computing node can refer to an independent server unit participating in multimodal model training, serving as a carrier of physical computing resources. Graphical memory (GPU) can refer to high-bandwidth memory (HBM) on an accelerator card (such as a GPU / NPU). Main memory (RAM) can refer to the host main memory (DRAM) of the computing node. Mapping to the virtual address space can refer to the unified conversion and inclusion of the physical GPU and RAM addresses scattered across various computing nodes into a globally unique, contiguous 64-bit global virtual address (GVA) space through memory registration technology. The virtual address space can refer to a set of logical addresses managed by a globally unified memory pool.

[0071] For example, on each compute node, by using the interface that registers (locks) the virtual address range of system memory (CPU memory) as page-locked memory (cuda Host Register) or registers a memory region (ibv_reg_mr) with a remote direct access device, the HBM physical address of each GPU is registered to the RDMA context to obtain remote access permission.

[0072] Initialize the storage pool and divide the HBM region GVA range into [0x100000000000, 0x140000000000]. Using the formula [start + i × max_size, start + (i + 1) × max_size], allocate a contiguous 10GB GVA segment to each of the 8 GPUs to obtain the virtual address space. For example, the virtual address of node 1-GPU0 is [0x100000000000, 0x1000000000000 + 10GB). Then, use the virtual addresses in the virtual address space as addressing identifiers.

[0073] In traditional technologies, the encoder needs to transmit hundreds of megabytes of sample embedding representations to the decoder via TCP / IP, resulting in high communication latency, consuming a large amount of high-speed bandwidth for interconnecting peripheral components, and being serial to computation. The embodiments of this disclosure transform the transmission of large tensor sample embedding representations into the transmission of lightweight metadata (GVA) through mapping relationships and addressing identifiers. The underlying remote access transmission can be executed in parallel with decoder computation, achieving communication-computation overlap and avoiding data transmission blocking the training pipeline.

[0074] According to embodiments of this disclosure, multiple sample embedding representations obtained by encoding multimodal samples using an encoder are written into a storage pool, and addressing identifiers indicating the storage location of each sample embedding representation in the storage pool are written into the sample pool. This includes: when multiple sample embedding representations are encapsulated into tensor objects using a tensor management tool and addressing identifiers are assigned to each tensor object, the sample embedding representations are written into the physical address corresponding to the addressing identifier in the storage pool, and the addressing identifiers are written into the sample pool. The encoder continues to process subsequent multimodal samples after writing the sample embedding representations into the storage pool, without waiting for the decoder to complete the reading operation of the sample embedding representations.

[0075] In the embodiments of this disclosure, the tensor management tool refers to the underlying tensor system, which is used for tensor encapsulation, addressing identifier allocation, and cross-process transfer management. A tensor object can be a sample embedding representation encapsulated by the tensor management tool, accompanied by metadata (e.g., shape, data type), and can be uniformly scheduled by the system. The physical address can refer to the actual storage location of the sample embedding representation in the computer hardware, including HBM offsets within a node, DRAM host virtual addresses, or remote memory addresses across nodes.

[0076] For example, the tensor management tool is invoked to encapsulate the embedding representation of each sample into a tensor object, and a unique 64-bit global virtual address is assigned to each tensor object. This GVA corresponds to the HBM physical address of a GPU on node A in the storage pool. The encoder writes the sample embedding representation to the HBM physical address in the storage pool pointed to by the GVA through the put interface of the tensor management tool, and asynchronously sends metadata such as GVA, sequence length, and label to the sample pool of the pure asynchronous scheduler. After the encoder completes writing the sample embedding representation, it returns immediately and continues to process the next batch of multimodal samples without waiting for the decoder to read them.

[0077] Figure 3B The diagram illustrates the global virtual address space distribution in a storage pool according to an embodiment of the present disclosure.

[0078] Each storage location is assigned a unique 64-bit Global Virtual Address (GVA), such as... Figure 3BAs shown, the storage pool reserves two non-overlapping contiguous regions, corresponding to high-bandwidth memory region 31 and dynamic random access memory region 32, respectively. Each region provides a GVA interval of, for example, [start, start + world_size × max_size], where start represents the globally agreed address base address, world_size represents the total number of processes (in a split-process scenario, the total number of processes can be the total number of cards occupied by the encoder and decoder), and max_size can be the maximum memory capacity contributed by a single process. The storage pool can be linearly divided among processes and is visible to all processes (e.g., process 1 and process 2). The GVA segments for process 1 and process 2 are [start + i × max_size, start + (i + 1) × max_size], where i is the index number of world_size.

[0079] Understandably, since each process has global visibility over all GVA regions (including remote HBM and DRAM), compute nodes do not need to be aware of the physical location of data when reading it. The underlying hardware and drivers can automatically translate the GVA into the corresponding physical address and initiate remote direct memory access directly via the high-speed bus. Data flows directly between the sender's physical memory and the receiver's physical memory, avoiding CPU intervention and unnecessary memory copying. This not only frees up CPU resources but also makes full use of the bandwidth of the underlying network.

[0080] Figure 3C The illustration shows a schematic diagram of an asynchronous data flow between an encoder and a decoder based on a storage pool, according to an embodiment of the present disclosure.

[0081] like Figure 3C As shown, encoder 101 contains multiple data computation node units (nodes 0 to 4), each processing different multimodal samples. DP0 generates sample embedding representation a, DP1 generates sample embedding representations b and f, DP2 generates sample embedding representations c and g, and so on. Each DP unit works independently without waiting for other DP units to complete. For example, after DP0 generates a, it immediately writes the sample embedding representation a to the storage pool via a write operation, and then returns to process the next input without waiting for DP1 or other units.

[0082] Unified Memory Pool 102 is a shared buffer between encoder 101 and decoder 104. It stores data written by the producer (encoder) (from left to right, the memory pool represents sample embeddings e, d, c, b, i, a, ...). Each write operation stores a local result (e.g., sample embeddings a, b, c) into the memory pool, identified by addressing (e.g., id1:a+d indicates that the data corresponding to id1 consists of sample embeddings a and d). Decoder 104, acting as a data consumer, actively reads data from memory pool 102 through id-get operations: get id1 retrieves sample embedding a+d; get id2 retrieves sample embedding b+c.

[0083] It is understandable that the producer (encoder) can embed the extracted samples into the storage pool in a non-blocking manner, and the consumer (decoder) can retrieve them by address identifier (get id) and access the storage transparently, thus forming a continuous producer-consumer data stream that is not constrained by batch boundaries.

[0084] According to embodiments of this disclosure, in traditional split training of multimodal models, the encoder must wait for all data tensor copies to finish processing the current batch of data before sending the data to the decoder, and the decoder must also wait for the entire batch of data to complete before starting computation, resulting in resource overhead due to bidirectional waiting between the encoder and decoder. Embodiments of this disclosure decouple the encoder and decoder data at batch boundaries by asynchronously passing storage pool buffers and addressing identifiers. The encoder writes and returns immediately, and the decoder retrieves data as needed, avoiding mutual blocking between heterogeneous modules.

[0085] According to embodiments of this disclosure, based on an addressing identifier, a scheduler reads a sample embedding representation corresponding to the addressing identifier from a storage pool, including: the scheduler, which contains the sample pool, reads the addressing identifier from the sample pool, the addressing identifier containing the virtual address of the sample embedding representation in the storage pool; the scheduler queries a global page table in the storage pool based on the virtual address to obtain the physical address of the sample embedding representation in the physical storage medium, the global page table indicating the mapping relationship between virtual addresses and physical addresses across nodes in each training process; the scheduler selects a transmission path based on the physical address of the sample embedding representation and the physical location of the current training process of the asynchronous scheduler; and the scheduler reads the sample embedding representation from the physical address through the transmission path to package the read sample embedding.

[0086] In the embodiments of this disclosure, the global page table can be a local data structure stored in each training process, recording the mapping relationship between the virtual address GVA and the physical address (the actual physical address of HBM / DRAM). It is initialized by each process after exporting the memory mapping information before training. The physical storage medium can refer to the hardware medium that actually stores the embedded representation of the samples, including HBM and DRAM within the node, as well as remote HBM / DRAM across nodes.

[0087] The transmission path can be a communication link automatically selected by the underlying transmission engine based on the location and medium type of the source and destination, including high-speed interconnects within nodes and remote direct memory access across nodes. Packaging processing can refer to a computationally cost-aware balanced packaging process, including Segment Tree Best-Fit Packing, execution cost modeling, two-dimensional load balancing and Zig-Zag Dispatching, which concatenates multiple variable-length samples into balanced micro-batches.

[0088] For example, the sample pool asynchronously receives metadata sent by the encoder. This metadata includes the virtual address of the sample embedding representation (e.g., 0x200000000000, belonging to the GVA of the HBM region). The Pack Actor in the scheduler pulls the GVA from the sample pool, queries the local global page table, and translates it to obtain the physical address: node A-GPU3-HBM-0x15000000. The Pack Actor determines that the current training process is located on node B, and the physical address belongs to the cross-node HBM, automatically selecting RDMA as the transmission path. It reads the sample embedding representation from the GPU3 HBM on node A via RDMA, packages multiple sample embedding representations into micro-batches / data packets based on the segment tree optimal packing algorithm, and then distributes them interleaved to multiple computing nodes in the decoder.

[0089] Traditional load balancing methods often measure load based on the original sequence length, which is difficult to adapt to the actual computation time after packaging. The embodiments of this disclosure utilize a scheduler that reads sample embeddings from the storage pool via addressing identifiers. Then, based on a computation cost-aware load balancing packaging algorithm (combining the actual time consumption characteristics of the FFN linear layer and the Attention operator), load balancing is performed by calculating the cost of data packets rather than the sequence length. This reduces padding probability, staggers the distribution of data, and balances the load among computing nodes, making the computational load of each node more consistent. This avoids individual slow nodes slowing down the overall training progress and improves the overall training efficiency of the model.

[0090] According to embodiments of this disclosure, the storage pool maintains a global page table composed of local page tables of each training process; the method further includes: allocating a plurality of consecutive virtual addresses to each training process; allocating physical addresses corresponding to the plurality of virtual addresses from the storage pool to obtain a mapping relationship between virtual addresses and physical addresses; synchronizing the mapping relationship of each training process to other processes, and updating the local page tables in other processes used to store the mapping relationship between virtual addresses and physical addresses of each process, so as to update the global page table based on the updated local page tables.

[0091] In the embodiments of this disclosure, a training process can refer to an independent execution unit participating in the training of a multimodal model, including encoder processes and decoder processes. Each process can be bound to one or more accelerator cards (GPUs / NPUs). A local page table can be a mapping data structure stored within a single training process, recording the correspondence between virtual addresses (GVAs) and physical addresses (local / remote HBM / DRAM addresses) from the perspective of that process. A global page table can be a logically distributed mapping set composed of the local page tables of all training processes. The consistency between the global page table and the local page table can be maintained through the synchronous update of the local page tables of each process, making it visible and consistent to all processes.

[0092] Synchronization to other processes can refer to the operation in which each training process, after completing local address mapping, sends its own mapping information (GVA range, physical address descriptor, etc.) to all other processes via network broadcast or point-to-point.

[0093] For example, the memory pool maintains page tables mapping virtual addresses (GVAs) to HBM / DRAM physical addresses used by each training process. During the memory pool initialization phase before training, each training process reserves a contiguous GVA region, allocates physical memory from the local operating system, and binds it to the reserved GVA region. Subsequently, each process exports its own memory mapping information to all other processes, and each process updates its local page table accordingly. Thus, each process can complete the mapping from virtual addresses to physical addresses locally, allowing each process to access remote memory transparently, just as if it were accessing local memory.

[0094] For example, during storage pool initialization, a GVA address base address start is allocated to the HBM region, denoted as 0x100000000000. A contiguous GVA segment is allocated to each process according to the formula [start+i×max_size, start+(i+1)×max_size] (for example, process 0 gets [0x100000000000, 0x100000000000+10GB)).

[0095] Process 0 (node ​​A-GPU0) allocates 10GB of HBM physical memory from the local operating system and establishes a mapping: GVA 0x100000001234 → Node A-GPU0 HBM physical offset 0x1234, which is recorded in process 0's local page table. Process 0 sends its own mapping information (GVA range, physical address descriptor, access permissions) to processes 8-15 (the decoder's training processes). After receiving the mapping information, process 8 updates its own local page table, adding the entry "GVA 0x100000001234 → Node A-GPU0 HBM physical offset 0x1234". Similarly, all decoder processes' local page tables contain this mapping to update the global page table based on the updated local page table.

[0096] In traditional static address mapping, the mapping relationship is fixed after memory allocation, making it difficult to handle scenarios where sample lengths change dynamically during training. In the embodiments of this disclosure, the mapping relationship between virtual and physical addresses is established once during the initialization phase. Subsequent data writing only needs to reuse the already mapped GVA space, eliminating the need for frequent page table updates. Combined with unified management of the storage pool, the system can dynamically adjust the storage distribution of HBM / DRAM based on data popularity (e.g., hot data is stored in HBM, cold data in DRAM), while upper-layer applications accessing the system via GVA remain unaffected. This achieves elastic scheduling of memory resources and avoids memory fragmentation and waste caused by static allocation.

[0097] According to embodiments of this disclosure, the data scheduling method for multimodal model training further includes: pre-running the encoder and decoder respectively to obtain a first throughput of the encoder and a second throughput of the decoder; determining a first number of encoder cards and a second number of decoder cards based on the ratio of the first throughput to the second throughput, so that the amount of data produced by the encoder per unit time matches the amount of data consumed by the decoder; and independently deploying a first device group of the encoder and a second device group of the decoder based on the first number of encoder cards and the second number of decoder cards respectively.

[0098] In embodiments of this disclosure, pre-run refers to a lightweight performance benchmark test performed on the encoder and decoder using a small number of real samples before formal training begins, used to collect processing capability metrics under steady-state conditions. The first throughput can refer to the processing capability per unit time measured by the encoder during the pre-run phase. The second throughput can refer to the processing capability per unit time measured by the decoder during the pre-run phase.

[0099] The first number of cards can refer to the number of computing devices allocated to the encoder, determined by the ratio of the first throughput to the second throughput, ensuring that the encoder's output rate matches the decoder's consumption rate. The second number of cards can refer to the number of computing devices allocated to the decoder, determined by the ratio of the first throughput to the second throughput, ensuring that the decoder's consumption rate matches the encoder's output rate. The first device group can consist of computing devices corresponding to the first number of cards, independently deploying encoder processes and employing the optimal parallelism strategy suitable for the encoder (e.g., pure data parallelism). The second device group can consist of computing devices corresponding to the second number of cards, independently deploying decoder processes and employing the optimal parallelism strategy suitable for the decoder (e.g., a combination of tensor parallelism and pipelined parallelism).

[0100] For example, before training, the encoder and decoder are pre-runned to measure their respective throughput (tokens / s / card). Based on this, the resource allocation ratio between the two is determined, ensuring that all tokens produced by the encoder per unit time are consumed by the decoder. For example, if the measured throughput encoder:decoder is approximately 10:1, then the resource allocation ratio (number of cards) encoder:decoder = 1:10. Furthermore, based on the first and second number of cards, the first device group of the encoder and the second device group of the decoder are deployed independently.

[0101] According to embodiments of this disclosure, in conventional technologies, the uneven distribution of encoder and decoder cards can easily lead to ineffective waiting times, such as data production occurring too quickly (waiting to be consumed) or too slowly (waiting to be produced). For example, conventional solutions often use a fixed ratio (e.g., 1:8 or 1:16) of empirically allocated cards, without considering model structure, input sequence length, and hardware differences. When the encoder is too fast, the storage pool may overflow due to data accumulation; when the decoder is too fast, it may become idle due to waiting for data. Embodiments of this disclosure achieve "on-demand allocation" by pre-running and measuring throughput, ensuring that the amount of data produced by the encoder per unit time is exactly consumed by the decoder, thus eliminating resource idleness or data backlog caused by rate mismatch at its source.

[0102] According to embodiments of this disclosure, the data scheduling method for multimodal model training further includes: the decoder initiating a storage access request for data packets in the storage pool through an addressing identifier; selecting a transmission path that matches the storage location and the physical location of the device group where the decoder is located, based on the storage location in the addressing identifier and the physical location of the device group where the decoder is located, and transmitting the data packets to the decoder.

[0103] In embodiments of this disclosure, a memory access request may refer to a memory access operation initiated by the decoder process based on the addressing identifier (GVA), and the request content is to read the sample embedding representation (data packet) at the corresponding address in the storage pool. The data packet may refer to the sample embedding representation stored in the storage pool or a micro-batch obtained by aggregating sample embedding representations.

[0104] For example, the decoder (node ​​B-GPU2) retrieves metadata from the sample pool and extracts the addressing identifier GVA0x100000001234. The decoder queries its local page table and resolves the storage location based on the GVA: node A, GPU0, HBM, offset 0x1234. The decoder detects that the storage location (node ​​A) and its own physical location (node ​​B) are across nodes, and the medium is HBM. The underlying transport engine automatically selects RDMA (Remote Direct Memory Access) as the transport path.

[0105] In traditional distributed training, regardless of whether the data is local or remote, a universal communication technology based on Transmission Control Protocol (TCP) or unlimited bandwidth technology is used, making it difficult to optimize for specific hardware topologies. The embodiments disclosed in this paper achieve precise matching of transmission paths through storage location and physical location: high-bandwidth dedicated lines are prioritized within nodes, and remote direct access technology is automatically enabled across nodes to bypass CPU bottlenecks. This ensures that each data transmission follows the shortest path under the current topology, reducing data supply latency during model training.

[0106] According to embodiments of this disclosure, selecting a transmission path that matches the storage location and the physical location includes: when the storage location and the physical location are different devices in the same node, using an intra-node high-speed interconnect transmission path as the transmission path; and when the storage location and the physical location are across nodes, using a remote direct memory access path as the transmission path.

[0107] In the embodiments of this disclosure, different devices within the same node can refer to different acceleration devices located within the same node but with storage and physical locations belonging to the same independent computing server (node) (e.g., GPU0 HBM stored on node 1, and GPU3 HBM physically located on node 1) or acceleration devices and host memory. High-speed interconnect transmission paths within a node can refer to dedicated high-speed communication links between acceleration devices within the node. Remote direct memory access paths can refer to cross-node transmission links based on RDMA (Remote Direct Memory Access) technology.

[0108] For example, the underlying layer automatically selects an available transmission engine based on the location between the source and destination of the data transmission request, as well as the memory type. If the data transmission request crosses memory levels within a node, the high-speed interconnect transmission path within the node is used as the transmission path; if the destination crosses nodes, a transmission endpoint is established (or reused) and distributed via a remote direct memory access path. Path selection is transparent to the upper layers, decoupling the hardware-related addressing and transmission details.

[0109] It is understandable that, based on the relative positions between the source and destination, different transmission paths are taken. The decoder does not need to be aware of the actual storage location of the micro-batch (data packet), but only operates on the GVA area and accesses the remote memory as if it were local memory.

[0110] Traditional distributed training typically employs a unified transmission protocol, either using the entire TCP / IP protocol stack with all data processed by the kernel and relayed by the CPU, or using a universal IB Verbs interface without distinguishing between topology scenarios, resulting in data transfers between devices on the same node detours through the network card or CPU. The embodiments disclosed in this paper customize transmission paths for two core topology scenarios: dedicated intra-node interconnects are prioritized for cross-device communication within the same node; remote direct access is used for cross-node communication, avoiding the memory copy overhead of traditional transmission paths, preventing unnecessary transmission detours, and maximizing the utilization of hardware topology capabilities.

[0111] According to embodiments of this disclosure, the data scheduling method for multimodal model training further includes: using a scheduler to obtain metadata provided by the encoder and storing it in a sample pool, the metadata including an address identifier and a sequence length; when the number of metadata in the sample pool is detected to be greater than a number threshold, or when the processing status of the decoder indicates that there are currently no data packets to be processed, the metadata in the sample pool is packaged to obtain multiple data packets, and the multiple data packets are sent to the decoder for processing.

[0112] In the embodiments of this disclosure, the metadata can be lightweight descriptive information generated by the encoder, which does not contain entity data of the sample embedding representation. It may include an addressing identifier (GVA, Global Virtual Address, used to locate the sample embedding representation in the storage pool); a sequence length (seq_len, the original sequence length corresponding to the sample embedding representation, such as the number of image patches or text tokens); and may also include auxiliary information such as labels and attention_mask. The number threshold can be used to determine whether the sample pool has accumulated enough metadata to trigger an efficient packaging operation.

[0113] The decoder's processing status refers to its current computational resource usage. A key indicator is whether there are no pending data packets (i.e., the decoder's computational units are idle and waiting for input data). This status can be obtained by the scheduler through heartbeat detection or querying the global batch dictionary. A data packet can refer to a micro-batch generated after packaging, containing a set of addressing identifiers (GVAs) for sample embedding representations; corresponding metadata such as sequence length and labels; and cost information such as the total number of tokens after packaging and the maximum sequence length.

[0114] Figure 4 A schematic diagram of the structure of a scheduler according to an embodiment of the present disclosure is shown.

[0115] like Figure 4 As shown, the scheduler 103 mainly includes two interfaces: a first interface for receiving output data from the encoder 101, and a second interface for providing training data to the decoder 104. The scheduler 103 can work collaboratively with components such as a sample pool (metapool) 103a, a pack actor (Pack Actor) 103b, a dispatch actor (Dispatch) 103c, and a global batch dictionary (global batch_dict) 103d: the sample pool 103a asynchronously receives the metadata 101a (including addressing identifier, input ID, label, sequence length, etc.) produced by the encoder 101 and generates a data snapshot; thereby determining the packing time based on the triggering strategy, which avoids invalid packing when there are insufficient samples and dynamically adjusts the data when the decoder has no data to process.

[0116] Packaging executor 103b performs computation cost-aware balanced packaging as an independent distributed computing unit; distribution executor 103c distributes micro-batches evenly to multiple computing nodes in the decoder according to computation cost; global batch dictionary 103d provides data to the consumer side (encoder) with data packet (dp_rank, step) as key, ensuring that multiple computing nodes obtain data asynchronously with logical correctness and without blocking each other.

[0117] It is understandable that the encoder only performs the forward process and puts the data into the sample pool, without waiting for the decoder to process it or for synchronization between the encoder's data computation nodes; the operation of putting data into the sample pool is asynchronous, achieving a completely non-blocking forward process. The decoder can achieve the overlap of data addressing and model computation simply by using asynchronous thread prefetching.

[0118] In one feasible embodiment, the scheduler 103 can also obtain the data consumption rate of the decoder 104 in real time and dynamically adjust the output of the encoder 101 (e.g., visual encoder ViT) to form a closed-loop balance between data production and data consumption.

[0119] According to embodiments of this disclosure, image sequence lengths in multimodal data often exhibit a long-tail distribution (a large number of short sequences and a small number of extremely long short sequences). Traditional fixed-batch strategies, when dealing with long-tail data, either block the entire batch by waiting for a few long sequences (increasing latency) or waste computational power due to excessive data padding. Embodiments of this disclosure utilize multiple triggering mechanisms to generate data packets / micro-batches promptly when samples are insufficient, avoiding global blocking caused by waiting for long-tail samples. Simultaneously, computationally cost-aware packaging ensures high computational density in the generated micro-batches, reducing data padding and increasing the effective computational power ratio.

[0120] According to embodiments of this disclosure, metadata in a sample pool is packaged to obtain multiple data packets, including: constructing a segment tree containing a preset integer range based on the sequence length of each of the multiple sequences in the metadata; the segment tree is used to manage multiple candidate packaged sequences; each leaf node in the segment tree is a packaging processing node in the scheduler used to process the sequences in the metadata; when a leaf node matching the resource required for the length of each candidate packaged sequence is found in the segment tree, the candidate packaged sequence is packaged using the leaf node to obtain multiple data packets, which are then sent to the decoder for processing.

[0121] In the embodiments of this disclosure, the sequence length can refer to the total length of tokens in a single multimodal sample, such as the number of batches after visual encoding of an image or the number of tokens after word segmentation of text, which can be used to quantify the computational resource requirements of the sample. The preset integer range can refer to the coverage range of the segment tree, denoted as [0, L], where L is the maximum micro-batch sequence length, corresponding to the maximum total number of tokens that a single micro-batch can hold. The segment tree can refer to the binary tree data structure maintained by the Pack Actor in the scheduler, where internal nodes store the maximum remaining capacity within the subtree range, and leaf nodes correspond to the remaining available capacity of candidate packing sequences (i.e., candidate micro-batches / Pack bins), used to manage the resource reserves of multiple candidate micro-batches.

[0122] Candidate packing sequences can be candidate micro-batches, which are intermediate packing units that have not yet reached their capacity limit. Each candidate packing sequence corresponds to a leaf node in the segment tree, storing the appended sample metadata and the current remaining capacity. Leaf nodes can be terminal nodes in the segment tree, each uniquely corresponding to a candidate packing sequence, storing the current remaining capacity of that candidate sequence, and are the operation unit for performing packing, directly managed by the packing execution body.

[0123] For example, the scheduler maintains a segment tree covering the integer interval [0, L]. Each leaf node corresponds to the remaining capacity of a candidate micro-batch, and each internal node stores the maximum remaining capacity within its subtree. For each incoming sequence s of length ℓ(s), the scheduler performs an O(logL) traversal: first, it checks the left subtree representing the smaller remaining capacity; if its maximum remaining capacity is ≥ ℓ(s), it recursively enters the left subtree; otherwise, it enters the right subtree, thus finding the micro-batch with the smallest remaining capacity that can accommodate the incoming sequence s with the best-fit. If found, the incoming sequence is appended to the micro-batch and its remaining capacity is updated; otherwise, a new micro-batch with a remaining capacity of L−ℓ(s) is created and inserted into the segment tree. When the remaining capacity of a micro-batch is lower than the minimum sequence length of the current data distribution, it is removed from the segment tree.

[0124] Traditional batching methods require waiting for all samples to be ready before sorting and concatenating them, and this process is performed serially with the encoder's forward pass and decoder's computation. This embodiment supports dynamic updates via segment trees, allowing for real-time matching node lookup without waiting for the entire batch of samples. The batching operation is performed asynchronously by an independent batching executor, completely overlapping with the encoder's forward pass and decoder's computation. This avoids pipeline bubbles caused by traditional batch processing and improves the overall efficiency of model training.

[0125] According to embodiments of this disclosure, the data scheduling method for multimodal model training further includes: determining the execution cost of each packet processing node for packaging candidate packet sequences, the execution cost including a first component and a second component, the first component being positively correlated with the total number of valid tokens in all sample embeddings within a micro-batch, and the second component being positively correlated with the upper limit of the sequence length of each sample embedding within the packet processing node; clustering packet processing nodes with similar execution costs into an array of the same cost level based on the execution cost of each packet processing node, so that the execution delay of packet processing nodes in the array of the same cost level remains consistent; arranging each data packet in descending order of execution cost in the array of the same cost level, and sequentially allocating each data packet to each computing node of the decoder for processing using an alternating reverse traversal method, so that the execution cost of each computing node is consistent.

[0126] In embodiments of this disclosure, the packet processing nodes can be leaf nodes in a segment tree. Each node can correspond to a candidate packet sequence (Pack bin), which, in physical implementation, corresponds to a logical unit in the scheduler managed by the packet execution body for storing metadata to be packetized. The packet cost can refer to the estimated computation time required to complete the forward and backward propagation of a micro-batch (data packet). The first component and the second component can refer to the linear and quadratic terms in the cost model used to calculate the packet cost, respectively.

[0127] A cost level array can refer to a discrete set divided according to the range of execution cost values. Clustered processing nodes with similar execution costs can be grouped into the same array (Bucket), with different arrays representing different computational intensity levels. Alternating reverse traversal can be a load-balanced distribution strategy.

[0128] For example, time consumption tests were conducted on the two types of operators (FFN / linear layer and variable-length attention layer) that account for the largest proportion of decoder time consumption. The results showed that the time consumption of FFN / linear layer is approximately linearly related to the total number of valid tokens in the packet; the time consumption of variable-length attention layer is mainly affected by the length of the longest subsequence in the packet, max_len, and is approximately proportional to the square of max_len. Based on this, in this embodiment of the disclosure, the execution cost of each packaged micro-batch (Packbin) P is modeled as shown in the following formula (1):

[0129] Cost(P)=α·Σs∈Plen(s)+β·(maxs∈Plen(s))² (1);

[0130] The first term (TotalTokens) represents the sum of the true lengths of all sequences within the bag, and α represents the time coefficients of the FFN and linear layers in the cost model; the second term (Attention Block) represents the square of the length of the longest sequence within the bag, and β represents the time coefficient of the Attention layer in the model.

[0131] Traditional load balancing allocates data solely based on sequence length or the total number of tokens, ignoring the complexity of attention layer operators in models (such as Transformers). The embodiments of this disclosure introduce a second component (the square of the longest sequence length) to capture the nonlinear computational surge caused by long sequence samples. This transforms load balancing from a simple data volume balancing process into a true computational volume balancing process, solving the traditional problem of a single long sequence slowing down the entire batch of data in conventional technologies.

[0132] According to embodiments of this disclosure, multiple data packets obtained from the packetized sample embedding representation are sent to the decoder for processing to obtain processing results. The process includes: using a packetization tool associated with the scheduler to read the corresponding sample embedding representation from the storage pool according to the addressing identifier, and performing packetization processing to generate multiple data packets; sending the multiple data packets to each computing node of the decoder for decoding processing to obtain the processing results of multimodal samples; wherein the write operation of the sample pool and the read operation of the packetization tool are executed asynchronously in parallel.

[0133] In the embodiments of this disclosure, the packaging tool, also known as the packaging executor, is the packaging actor in the scheduler. It is an independent, stateful, concurrent unit implemented based on an independent distributed computing framework, specifically responsible for performing computationally cost-aware balanced packaging operations. The write operation refers to the operation where the encoder (such as ViT) asynchronously writes the corresponding metadata to the sample pool after completing the forward encoding of a single sample / micro-batch. After writing, the encoder immediately returns to process subsequent samples without waiting for a response from the downstream module. The read operation refers to the operation where the packaging tool asynchronously reads metadata from the sample pool, queries the global page table based on the addressing identifier in the metadata to obtain the physical address, and reads the corresponding sample embedding representation from the storage pool. Execution is activated when the triggering strategy meets the conditions.

[0134] Decoding can refer to the complete training process where the decoder on the computing node reads the corresponding sample embedding representation from the storage pool based on the addressing identifier in the data packet, performs forward inference to generate hidden states, and then performs backpropagation to generate gradients. The processing results may include, but are not limited to, loss values, gradient information, prediction results, attention weights, etc., and are the core inputs for model iterative updates.

[0135] For example, the packaging execution unit can first sort the metadata in the sample pool by sequence length (seq_len), select multiple anchor samples (anchor_start to anchor_end) within the anchor width (anchor_width), and then perform best-fit packaging to generate multiple data packets in the sample pool. Then, it calculates the cost of each data packet according to the cost model mentioned above, and clusters micro-batches with similar computational intensity into the same discrete bucket to ensure that the execution latency of micro-batches / data packets entering the same pipeline is consistent (otherwise, micro-batches with huge cost differences entering the same pipeline will create pipeline bubbles). Within each bucket, the data packets are sorted in descending order of cost, and staggered distribution is used (alternating reverse traversal between computation node 0 → computation node _{N-1} and computation node _{N-1} → computation node 0) to distribute the micro-batches to the computation nodes of each data parallel computation in sequence, so that high-cost and low-cost data packets are paired on each computation node, forcing the total cost of each computation node to be similar, thereby achieving computational load balancing across data tensors and pipelines or inter-layer dimensions.

[0136] According to embodiments of this disclosure, the execution rhythms of the encoder, scheduler, and decoder are tightly coupled in related technologies, and performance fluctuations in any module will affect the overall process: if the packaging tool is temporarily busy, the encoder will be forced to block and wait, causing forward computation to stall. In the embodiments of this disclosure, the sample pool acts as an asynchronous buffer layer, allowing the encoder to continuously write and the packaging tool to read on demand. Even if the packaging tool is temporarily delayed due to a momentary high load, it will not block the forward execution of the encoder, thus improving the anti-interference capability and stability of large-scale training.

[0137] Figure 5 A block diagram of a data scheduling apparatus for multimodal model training according to an embodiment of the present disclosure is shown schematically.

[0138] like Figure 5 As shown, the multimodal model includes encoders and decoders deployed independently in different device groups. The data scheduling device 500 used in multimodal model training includes a writing module 501, a reading module 502, and a sending module 503.

[0139] The writing module 501 is used to write multiple sample embedding representations obtained by encoding multimodal samples using an encoder into a storage pool, and to write addressing identifiers indicating the storage location of each sample embedding representation in the storage pool into the sample pool. The amount of data in the addressing identifiers is less than the amount of data in the sample embedding representations. It can be understood that the writing module 501 can be used to perform operation S201 as described above, which will not be elaborated here.

[0140] The reading module 502 is used to read the sample embedding representation corresponding to the addressing identifier from the storage pool using the scheduler. It can be understood that the reading module 502 can be used to perform operation S202 as described above, which will not be elaborated upon here.

[0141] The sending module 503 is used to send multiple data packets obtained from the embedded representation of the packaged samples to the decoder for processing, and to obtain the processing result, so as to train a multimodal model based on the loss between the processing result and the true label. It can be understood that the sending module 503 can be used to perform operation S203 in the above method, which will not be elaborated here.

[0142] According to embodiments of this disclosure, the writing module 501, reading module 502, and sending module 503 in the data scheduling device 500 for multimodal model training write sample embedding representations generated by the encoder into a storage pool and store lightweight addressing identifiers in the sample pool. The scheduler asynchronously reads the corresponding sample embedding representations from the storage pool based on the addressing identifiers and packages them for the decoder, thereby achieving decoupled execution of the encoder and decoder in time. The encoder's "write-and-return" operation eliminates the need to wait for the decoder to finish processing, and the decoder's "on-demand retrieval" operation eliminates the need to wait for the encoder to complete the entire batch, avoiding batch-level synchronization barriers between traditional heterogeneous modules. Since the encoder and decoder are deployed independently in different device groups, they can each adopt the optimal parallelism, avoiding parallelism mismatch caused by module heterogeneity. In addition, the scheduler performs inter-process communication based on lightweight addressing identifiers rather than large tensor sample embedding representations, which greatly reduces the communication overhead between heterogeneous modules and improves the resource utilization and scalability of multimodal large language model training.

[0143] According to an embodiment of this disclosure, the writing module 501 includes: a writing submodule, used to write the sample embedding representation to the physical address corresponding to the addressing identifier in the storage pool and write the addressing identifier to the sample pool, respectively, when multiple sample embedding representations are encapsulated into tensor objects using tensor management tools and addressing identifiers are assigned to each tensor object; wherein, after the encoder writes the sample embedding representation to the storage pool, it continues to process subsequent multimodal samples without waiting for the decoder to complete the reading operation of the sample embedding representation.

[0144] According to embodiments of this disclosure, the reading module 502 includes: a first reading submodule, an acquisition submodule, a selection submodule, and a second reading submodule. The first reading submodule is used by a scheduler containing a sample pool to read an addressing identifier from the sample pool, the addressing identifier containing the virtual address of the sample embedding representation in the storage pool; the acquisition submodule is used to query a global page table in the storage pool based on the virtual address to obtain the physical address of the sample embedding representation in the physical storage medium, the global page table indicating the mapping relationship between virtual addresses and physical addresses across nodes in each training process; the selection submodule is used to select a transmission path based on the physical address of the sample embedding representation and the physical location of the current training process of the asynchronous scheduler; the second reading submodule is used to read the sample embedding representation from the physical address through the transmission path to perform packet processing on the read sample embedding.

[0145] According to embodiments of this disclosure, the storage pool maintains a global page table composed of local page tables of each training process; the apparatus further includes: an address allocation module, a relationship determination module, and an update module. The address allocation module is used to allocate multiple consecutive virtual addresses to each training process; the relationship determination module is used to allocate physical addresses corresponding to each of the multiple virtual addresses from the storage pool, obtaining a mapping relationship between virtual addresses and physical addresses; the update module is used to synchronize the mapping relationship of each training process to other processes, and update the local page tables in other processes that store the mapping relationship between virtual addresses and physical addresses of each process, so as to update the global page table based on the updated local page tables.

[0146] According to embodiments of this disclosure, the above-mentioned apparatus further includes: an operation module, a card number determination module, and a deployment module. The operation module is used to pre-run the encoder and decoder respectively to obtain a first throughput of the encoder and a second throughput of the decoder; the card number determination module is used to determine a first number of cards for the encoder and a second number of cards for the decoder based on the ratio of the first throughput to the second throughput, so that the amount of data produced by the encoder per unit time matches the amount of data consumed by the decoder; the deployment module is used to independently deploy a first device group for the encoder and a second device group for the decoder based on the first number of cards and the second number of cards.

[0147] According to embodiments of this disclosure, the apparatus further includes a mapping module and an application module. The mapping module is used to map the video memory or memory of each computing node on which the encoder and decoder are deployed to a virtual address space; the application module is used to use the virtual address in the virtual address space as an addressing identifier indicating the storage location of each sample embedding in the storage pool.

[0148] According to embodiments of this disclosure, the above apparatus further includes an initiation module and a path selection module. The initiation module is used by the decoder to initiate a memory access request for data packets in the storage pool through an addressing identifier; the path selection module is used to select a transmission path that matches the storage location and the physical location of the device group where the decoder is located, based on the storage location in the addressing identifier and the physical location of the device group where the decoder is located, and then transmit the data packets to the decoder.

[0149] According to embodiments of this disclosure, the path selection module includes: a first submodule and a second submodule. The first submodule is used to select the intra-node high-speed interconnect transmission path as the transmission path when the storage location and physical location are different devices within the same node; the second submodule is used to select the remote direct memory access path as the transmission path when the storage location and physical location are across nodes.

[0150] According to embodiments of this disclosure, the above apparatus further includes: a metadata acquisition module and a packaging module. The metadata acquisition module is used to acquire metadata provided by the encoder using a scheduler and store it in a sample pool. The metadata includes an address identifier and a sequence length. The packaging module is used to package the metadata in the sample pool into multiple data packets when it is detected that the number of metadata in the sample pool is greater than a threshold number, or when the processing status of the decoder indicates that there are currently no data packets to be processed. The multiple data packets are then sent to the decoder for processing.

[0151] According to embodiments of this disclosure, the packaging module includes a construction submodule and a packaging submodule. The construction submodule is used to construct a segment tree containing a preset integer range based on the sequence lengths of multiple sequences in the metadata. The segment tree is used to manage multiple candidate packaging sequences, and each leaf node in the segment tree is a packaging processing node in the scheduler used to process the sequences in the metadata. The packaging submodule is used to package the candidate packaging sequences using the leaf nodes when a leaf node matching the resource requirements of each candidate packaging sequence length is found in the segment tree, obtaining multiple data packets, which are then sent to the decoder for processing.

[0152] According to embodiments of this disclosure, the above-mentioned apparatus further includes: a cost determination module, a clustering module, and a data packet allocation module. The cost determination module is used to determine the execution cost of each packet processing node for packaging candidate packet sequences. The execution cost includes a first component and a second component. The first component is positively correlated with the total number of valid tokens in all sample embeddings within a micro-batch, and the second component is positively correlated with the upper limit of the sequence length of each sample embedding within the packet processing node. The clustering module clusters packet processing nodes with similar execution costs into an array of the same cost level, based on the execution costs of each packet processing node, so that the execution delay duration of packet processing nodes in the same cost level array remains consistent. The data packet allocation module arranges the data packets in the array of the same cost level in descending order of execution cost and allocates each data packet sequentially to each computing node of the decoder for processing using an alternating reverse traversal method, so that the execution costs of each computing node are consistent.

[0153] According to embodiments of this disclosure, the sending module 503 includes a data packet generation submodule and a decoding submodule. The data packet generation submodule is used to read the corresponding sample embedding representation from the storage pool according to the addressing identifier using a packing tool associated with the scheduler, and perform packing processing to generate multiple data packets. The decoding submodule is used to send the multiple data packets to each computing node of the decoder for decoding processing to obtain the processing result of the multimodal samples. The write operation to the sample pool and the read operation of the packing tool are executed asynchronously and in parallel.

[0154] Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure, or at least part of the functions of any one or more of them, can be implemented in one module. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be implemented by dividing them into multiple modules. Any one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as hardware circuitry, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a System-on-Chip, a System-on-a-Substrate, a System-on-Package, an Application-Specific Integrated Circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, one or more of the modules, submodules, units, and subunits according to embodiments of the present disclosure can be at least partially implemented as computer program modules, which, when run, can perform corresponding functions.

[0155] For example, any plurality of the write module 501, read module 502, and send module 503 can be combined into one module / unit / subunit, or any one of these modules / units / subunits can be split into multiple modules / units / subunits. Alternatively, at least part of the functionality of one or more of these modules / units / subunits can be combined with at least part of the functionality of other modules / units / subunits and implemented in one module / unit / subunit. According to embodiments of this disclosure, at least one of the write module 501, read module 502, and send module 503 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or any other reasonable means of integrating or packaging the circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the write module 501, read module 502, and send module 503 may be implemented at least partially as a computer program module, which can perform corresponding functions when the computer program module is run.

[0156] Figure 6 A block diagram of an electronic device suitable for implementing the methods described above, according to embodiments of the present disclosure, is illustrated schematically. Figure 6 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0157] like Figure 6 As shown, an electronic device 600 according to an embodiment of this disclosure includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage portion 608 into a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of this disclosure.

[0158] RAM 603 stores various programs and data required for the operation of electronic device 600. Processor 601, ROM 602, and RAM 603 are interconnected via bus 604. Processor 601 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 602 and / or RAM 603. It should be noted that the programs may also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in said one or more memories.

[0159] According to embodiments of this disclosure, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to a bus 604. The system 600 may also include one or more of the following components connected to the input / output (I / O) interface 605: an input section 606 including a keyboard, mouse, etc.; an output section 607 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 608 including a hard disk, etc.; and a communication section 609 including a network interface card such as a LAN card, modem, etc. The communication section 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the input / output (I / O) interface 605 as needed. A removable medium 611, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 610 as needed so that computer programs read from it can be installed into the storage section 608 as needed.

[0160] According to embodiments of this disclosure, the method flow according to embodiments of this disclosure can be implemented as a computer software program. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable storage medium, the computer program containing program code for performing the methods shown in the flowchart. In such embodiments, the computer program can be downloaded and installed from a network via communication section 609, and / or installed from removable medium 611. When the computer program is executed by processor 601, it performs the functions defined in the system of embodiments of this disclosure. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0161] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0162] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium. Examples include, but are not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0163] For example, according to embodiments of this disclosure, a computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0164] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods provided in the embodiments of this disclosure. When the computer program product is run on an electronic device, the program code is used to enable the electronic device to implement the methods provided in the embodiments of this disclosure.

[0165] When the computer program is executed by the processor 601, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0166] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and downloaded and installed via the communication section 609, and / or installed from the removable medium 611. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0167] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0168] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions. Those skilled in the art will understand that the features recited in the various embodiments and / or claims of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not expressly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0169] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. The scope of this disclosure is defined by the appended claims and their equivalents. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A data scheduling method for training a multimodal model, the multimodal model comprising encoders and decoders independently deployed in different device groups, the method comprising: Multiple sample embedding representations obtained by encoding multimodal samples using the encoder are written into a storage pool, and an addressing identifier indicating the storage location of each sample embedding representation in the storage pool is written into the sample pool. The data size of the addressing identifier is smaller than the data size of the sample embedding representation. Based on the addressing identifier, the scheduler reads the sample embedding representation corresponding to the addressing identifier from the storage pool; The multiple data packets obtained by packaging the sample embedding representation are sent to the decoder for processing to obtain the processing result, and the multimodal model is trained based on the loss between the processing result and the real label.

2. The method according to claim 1, wherein, Multiple sample embedding representations obtained by encoding multimodal samples using the encoder are written into a storage pool, and an addressing identifier indicating the storage location of each sample embedding representation in the storage pool is written into the sample pool, including: When the multiple sample embedding representations are encapsulated into tensor objects using tensor management tools and an addressing identifier is assigned to each tensor object, the sample embedding representations are written into the physical address corresponding to the addressing identifier in the storage pool, and the addressing identifier is written into the sample pool. The encoder writes the sample embedding representation into the storage pool and then continues to process subsequent multimodal samples without waiting for the decoder to complete the reading operation of the sample embedding representation.

3. The method according to claim 1, wherein, Based on the addressing identifier, the scheduler reads the sample embedding representation corresponding to the addressing identifier from the storage pool, including: The addressing identifier, which contains the sample embedding, is read from the sample pool by the scheduler that includes the sample pool; The global page table in the storage pool is queried according to the virtual address to obtain the physical address of the sample embedding representation in the physical storage medium. The global page table is used to indicate the mapping relationship between the virtual address and the physical address across nodes in each training process. The transmission path is selected based on the physical address represented by the sample embedding and the physical location of the current training process of the asynchronous scheduler; The sample embedding representation is read from the physical address via the transmission path to package the read sample embedding.

4. The method according to claim 3, wherein, The storage pool maintains a global page table composed of the local page tables of each training process; the method further includes: Each training process is allocated a contiguous set of virtual addresses; Allocate physical addresses corresponding to each of the plurality of virtual addresses from the storage pool to obtain the mapping relationship between the virtual addresses and the physical addresses; The mapping relationship of each training process is synchronized to other processes, and the local page table in other processes, which stores the mapping relationship between virtual addresses and physical addresses of each process, is updated, so as to update the global page table based on the updated local page table.

5. The method according to claim 1, further comprising: The encoder and the decoder are pre-run to obtain a first throughput of the encoder and a second throughput of the decoder; Based on the ratio of the first throughput to the second throughput, the first number of cards for the encoder and the second number of cards for the decoder are determined so that the amount of data produced by the encoder per unit time matches the amount of data consumed by the decoder. Based on the first number of cards and the second number of cards, the first device group of the encoder and the second device group of the decoder are deployed independently.

6. The method according to claim 1, further comprising: The video memory or memory of each computing node, which is equipped with the encoder and the decoder, is mapped to the virtual address space. The virtual address in the virtual address space is used as an addressing identifier to indicate the storage location of each sample in the storage pool.

7. The method according to claim 1, further comprising: The decoder initiates a storage access request for data packets in the storage pool through the addressing identifier; Based on the storage location in the addressing identifier and the physical location of the device group where the decoder is located, a transmission path matching the storage location and the physical location is selected, and the data packet is transmitted to the decoder.

8. The method according to claim 7, wherein, Selecting a transmission path that matches the storage location and the physical location includes: When the storage location and the physical location are different devices in the same node, the high-speed interconnect transmission path within the node is used as the transmission path; In cases where the storage location and the physical location are on different nodes, the remote direct memory access path is used as the transmission path.

9. The method according to claim 1, further comprising: The scheduler obtains the metadata provided by the encoder and stores it in the sample pool. The metadata includes the addressing identifier and the sequence length. If the number of metadata in the sample pool is detected to be greater than the number threshold, or if the processing status of the decoder indicates that there are currently no data packets to be processed, the metadata in the sample pool is packaged to obtain the multiple data packets, and the multiple data packets are sent to the decoder for processing.

10. The method according to claim 9, wherein, The metadata in the sample pool is packaged to obtain the multiple data packets, including: Based on the sequence lengths of multiple sequences in the metadata, a segment tree containing a preset integer range is constructed. The segment tree is used to manage multiple candidate packing sequences, and each leaf node in the segment tree is a packing processing node in the scheduler used to process the sequences in the metadata. If a leaf node matching the resource required for the length of each candidate packet sequence is found in the segment tree, the candidate packet sequence is packaged using the leaf node to obtain the multiple data packets, which are then sent to the decoder for processing.

11. The method of claim 10, further comprising: The execution cost of each packing processing node for packing the candidate packing sequence is determined. The execution cost includes a first component and a second component. The first component is positively correlated with the total number of valid tokens in all sample embeddings within the micro-batch. The second component is positively correlated with the upper limit of the sequence length of each sample embedding within the packing processing node. Based on the execution cost of each packaging processing node, packaging processing nodes with similar execution costs are clustered into the same cost level array to ensure that the execution latency of packaging processing nodes in the same cost level array remains consistent. In the array of the same cost level, each data packet is arranged in descending order of execution cost, and each data packet is sequentially distributed to each computing node of the decoder for processing by an alternating reverse traversal method, so that the execution cost of each computing node is consistent.

12. The method according to claim 1, wherein, The multiple data packets obtained by packaging the sample embedding representation are sent to the decoder for processing to obtain the processing result, including: The packing tool associated with the scheduler reads the corresponding sample embedding representation from the storage pool according to the addressing identifier and performs packing processing to generate multiple data packets; The multiple data packets are sent to each computing node of the decoder for decoding processing to obtain the processing results of the multimodal samples; The write operation of the sample pool and the read operation of the packaging tool are executed asynchronously and in parallel.

13. A data scheduling apparatus for training a multimodal model, the multimodal model comprising encoders and decoders independently deployed in different device groups, the apparatus comprising: The writing module is used to write multiple sample embedding representations obtained by encoding multimodal samples using the encoder into a storage pool, and to write an addressing identifier indicating the storage location of each sample embedding representation in the storage pool into the sample pool, wherein the data size of the addressing identifier is smaller than the data size of the sample embedding representation; A reading module is used to read the sample embedding representation corresponding to the addressing identifier from the storage pool using a scheduler based on the addressing identifier; The sending module is used to send multiple data packets obtained by packaging the sample embedding representation to the decoder for processing, and to obtain the processing result, so as to train the multimodal model based on the loss between the processing result and the real label.

14. An electronic device comprising: One or more processors; Memory, used to store one or more programs. Wherein, when the one or more programs are executed by the one or more processors, the one or more processors implement the method of any one of claims 1 to 12.

15. A computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the method of any one of claims 1 to 12.

16. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1 to 12.

Images