Multi-modal data distributed training method and system
By preprocessing and allocating multimodal data, independently executing modality decisions, and utilizing global gradient fusion and All-Reduce operations, the problems of gradient disorder and memory crashes in multimodal data training are solved, thereby improving training efficiency and model stability.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XINGHAITU (SUZHOU) ARTIFICIAL INTELLIGENCE TECHNOLOGY CO LTD
- Filing Date
- 2026-06-12
- Publication Date
- 2026-07-14
AI Technical Summary
In existing technologies, multimodal data training suffers from problems such as gradient disorder, waste of computing resources, inconsistent motion spaces of multiple robot bodies, memory crashes and long breakpoint recovery times due to training large datasets with long trajectories.
By preprocessing multimodal data, separating different modal data and allocating them to the same computing node, independently executing modal decisions, selecting modes using probabilistic or deterministic decision-making methods, performing forward and backward propagation, and utilizing global gradient fusion and All-Reduce operations to achieve data matching and breakpoint recovery.
It effectively solves the problems of gradient disorder and waste of computing resources, improves training efficiency, reduces memory usage, ensures stable training of the model on different robot bodies, and avoids problems such as memory crashes and long breakpoint recovery times.
Smart Images
Figure CN122390005A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of robotics, and more specifically, to a multimodal data distributed training method and system. Background Technology
[0002] With the development of large-scale model technology, the field of embodied intelligence is shifting from single robot control strategies to general embodied base models based on large-scale vision-language-action (VLA). To enable models to possess strong generalization capabilities and zero-shot cross-scene execution, the exploration of scaling laws in the embodied domain is particularly important. Simultaneously, to maintain the language capabilities and visual understanding of pre-trained visual language models, joint training using massive amounts of visual language model (VLM) data (such as visual question answering (VQA)) and motion trajectory data from multi-body robots has become a mainstream research direction.
[0003] In existing technical solutions, most mainstream open-source frameworks are designed for training single-type robot bodies, or only support simple fine-tuning of VLM, or simply perform brute-force training by mixing different types of robots together, which has the following problems: 1. Mixed training with VLM and VLA data can easily lead to gradient disorder and inefficiency: Robot VLA data contains high-frequency action sequences and images from multiple perspectives, typically with long sequences (e.g., about 800 tokens); while VLM data (such as simple text-based question-and-answer sessions) usually has shorter sequences (e.g., less than 500 tokens). Due to the significant difference in sequence length, a large amount of padding operations must be performed on VLM data within the same batch to align tensor dimensions. These padding characters do not contain any useful information but consume a great deal of GPU memory and forward / backward propagation computational power (FLOPs), resulting in serious waste of computational resources. In addition, the optimization objective of VLM data is usually the cross-entropy of natural language, while VLA data involves action token regression and flow velocity loss in flow matching. The loss scale and convergence surface of the two are very different. If they are forcibly mixed for computation within the same batch, the gradients of the two different optimization directions will overlap and conflict within a single GPU, making it difficult for the optimizer to find the global optimum, leading to catastrophic forgetting or decreased accuracy of action output.
[0004] 2. Inconsistent motion spaces across multiple robots, leading to incompatibility: Different robot hardware (such as dual-arm, single-arm, robotic arms with different degrees of freedom, humanoids, etc.) possess completely different motion space dimensions. Existing brute-force padding results in the same motion dimension index representing entirely different physical meanings on different robots. For example, the 8th dimension of a vector (Dim 8) might be "left-hand gripper opening degree" on robot A, but becomes "forward movement speed of the chassis" on robot B. This forced mixed training with "same dimension, different semantics" causes the model to receive contradictory gradient signals (i.e., semantic anomalies) when optimizing that dimension, significantly slowing down convergence and even causing cross-body motion control to completely fail.
[0005] 3. Training with long-trajectory, large-scale data is highly susceptible to memory crashes: During the training startup phase, each computing node (Node) or process (Rank) in the cluster needs to load the entire dataset's metadata (such as file paths, lengths, ontology labels, etc. for all trajectories) into its local memory, and then allocate the data subsets it needs to read through the sampler. However, robot data is usually in units of trajectories, and a single trajectory typically contains thousands of high-resolution image sequences and high-frequency motion signals. The DataLoader stores many episodes in memory during the sampling process, which can easily lead to training interruptions due to memory overflow.
[0006] 4. Checkpoint Resume is time-consuming and extremely inefficient: When training is interrupted and restarted in a batch, standard samplers lack fine-grained state memory. To restore the system to the random state and data progress before the interruption, the system typically needs to iterate the DataLoader from scratch. Since the DataLoader is tied to physical data reading and data augmentation (Transforms) operations, the system must actually read all the pre-interrupted training data from disk and discard it. With the scale of pre-training data, this "idling" process can cause severe I / O bottlenecks, taking tens of minutes or even hours, resulting in expensive GPU computing power being idle.
[0007] To address the aforementioned issues, this application proposes a distributed training method and system for multimodal data. Through reasonable scheduling and optimization strategies, it solves the problems of gradient disorder, incompatibility of ontology data, and memory crashes that are prone to occur during training of mixed visual language data and multidimensional action data. Summary of the Invention
[0008] This invention provides a multimodal data distributed training method and system to overcome at least one technical problem existing in the prior art.
[0009] In a first aspect, embodiments of the present invention provide a multimodal data distributed training method, comprising: The original multimodal data is preprocessed to obtain multiple configuration data pools; Each configuration data pool is equally divided into N fragments, and fragments with the same index in different configuration data pools are assigned to the same computing node; where N represents the total number of computing nodes currently participating in training; Each computing node uses its assigned shard number as an index to load the corresponding shard data subset. Before each iteration begins, each computing node independently performs modal decision-making to obtain the modal decision-making results for this round of training; Based on the modality decision result, select the corresponding data loader to load the homogeneous batch training data of the corresponding modality from the data subset of that node; Each computing node sequentially performs forward and backward propagation on the homogeneous batch training data to obtain the gradient of each node; Global gradient fusion is performed on the gradients of all computing nodes to obtain the global gradient; All computing nodes use the global gradient to update model parameters.
[0010] Optionally, the raw multimodal data may be preprocessed, specifically including: The original multimodal data were classified according to data type to obtain the visual language understanding dataset and the visual language action dataset. The visual language understanding dataset and the visual language action dataset are preprocessed to obtain a visual language understanding configuration data pool and a visual language action configuration data pool, respectively.
[0011] Optionally, the visual language understanding dataset is preprocessed, specifically including: The modal data in the visual language understanding dataset are filled into a preset visual language understanding template, and the filled strings are parsed and segmented to obtain a visual language understanding token sequence; wherein, the modal data in the visual language understanding dataset includes at least image data and text question-answering data; The visual language understanding token sequence is unified into a semantic ontology pool to obtain a visual language understanding configuration data pool.
[0012] Optionally, the visual language action dataset is preprocessed, specifically including: Define a standard virtual ontology, which includes multiple semantic blocks divided by physical components, and assign a fixed maximum dimension to each semantic block; The original action vector is decomposed according to component type and filled into the corresponding semantic blocks. The filled semantic blocks are then spliced together in a predetermined physical order to form a unified action vector. The modal data and unified action vectors in the visual language action dataset are filled into a preset visual language action template; wherein, the modal data in the visual language action dataset includes at least image data and text instruction data; The padded string is parsed and segmented to obtain a visual language action token sequence; Based on the physical ontology type corresponding to the visual language action token sequence, the visual language action token sequence is divided into multiple visual language action configuration data pools.
[0013] Optionally, when the original motion vector is decomposed according to the component type and filled into the corresponding semantic block, if the current robot has the component, the motion data is filled into the corresponding semantic block. If the dimension is less than the maximum dimension of the semantic block, the filling is only done at the end of the semantic block. If the current robot does not have the component, the corresponding semantic block is filled with a preset mask value.
[0014] Optionally, modal decision-making is performed, specifically including: Modal decision-making can be performed using probabilistic decision-making methods or deterministic decision-making methods.
[0015] Optionally, the data types include visual language understanding data and visual language action data; Modal decision-making is performed using probabilistic decision-making methods, specifically including: If the sampling probability of visual language understanding data is set to p, then the sampling probability of visual language action data is 1-p; Each computing node generates a random number r, r∈[0,1], based on a uniform distribution. Compare r and p. If r < p, the corresponding computing node will perform the visual language understanding task in the current iteration step; otherwise, the corresponding computing node will perform the visual language action task in the current iteration step.
[0016] Optionally, based on the modality decision result, selecting the corresponding data loader to load homogeneous batch training data of the corresponding modality from the data subset of that node further includes: When training is interrupted, each data loader creates an iterator based on the interruption location. The iterator performs index jumps based on the fragment number and inner offset to locate the interruption location and continue reading data, thus realizing breakpoint resumption of training.
[0017] Secondly, the present invention also provides a multimodal data distributed training system, comprising: The preprocessing module is used to preprocess the raw multimodal data to obtain multiple configuration data pools; The partitioning module is used to equally partition each configuration data pool into N partitions and assign partitions with the same index in different configuration data pools to the same computing node; where N represents the total number of computing nodes currently participating in training; The loading module is used to enable each computing node to load the corresponding data subset of the shard based on the allocated shard number as an index. The decision module is used to enable each computing node to independently execute modal decisions before the start of each iteration, and obtain the modal decision results of this round of training; The filtering module is used to select the corresponding data loader from the data subset of the node to load the homogeneous batch training data of the corresponding mode based on the modality decision result. The execution module is used to enable each computing node to sequentially perform forward and backward propagation on the homogeneous batch training data to obtain the gradient of each node; The fusion module is used to perform global gradient fusion on the gradients of all computing nodes to obtain the global gradient; The update module is used to enable all computing nodes to update model parameters using the global gradient.
[0018] Optionally, the preprocessing module is specifically used for: The original multimodal data were classified according to data type to obtain the visual language understanding dataset and the visual language action dataset. The visual language understanding dataset and the visual language action dataset are preprocessed to obtain a visual language understanding configuration data pool and a visual language action configuration data pool, respectively.
[0019] Optionally, when preprocessing the visual language understanding dataset, the preprocessing module is specifically used for: The modal data in the visual language understanding dataset are filled into a preset visual language understanding template, and the filled strings are parsed and segmented to obtain a visual language understanding token sequence; wherein, the modal data in the visual language understanding dataset includes at least image data and text question-answering data; The visual language understanding token sequence is unified into a semantic ontology pool to obtain a visual language understanding configuration data pool.
[0020] Optionally, when preprocessing the visual language action dataset, the preprocessing module is specifically used for: Define a standard virtual ontology, which includes multiple semantic blocks divided by physical components, and assign a fixed maximum dimension to each semantic block; The original action vector is decomposed according to component type and filled into the corresponding semantic blocks. The filled semantic blocks are then spliced together in a predetermined physical order to form a unified action vector. The modal data and unified action vectors in the visual language action dataset are filled into a preset visual language action template; wherein, the modal data in the visual language action dataset includes at least image data and text instruction data; The padded string is parsed and segmented to obtain a visual language action token sequence; Based on the physical ontology type corresponding to the visual language action token sequence, the visual language action token sequence is divided into multiple visual language action configuration data pools.
[0021] Optionally, the decision module is specifically used for: Modal decision-making can be performed using probabilistic decision-making methods or deterministic decision-making methods.
[0022] Optionally, the data type includes visual language understanding data and visual language action data; when a probabilistic decision-making method is used to perform modal decision-making, the decision module is specifically used for: If the sampling probability of visual language understanding data is set to p, then the sampling probability of visual language action data is 1-p; Each computing node generates a random number r, r∈[0,1], based on a uniform distribution. Compare r and p. If r < p, the corresponding computing node will perform the visual language understanding task in the current iteration step; otherwise, the corresponding computing node will perform the visual language action task in the current iteration step.
[0023] Optionally, it also includes a positioning module, which is used to enable each data loader to create an iterator based on the interruption position when training is interrupted. The iterator performs index jump based on the fragment number and the inner offset to locate the interruption position and continue reading data, thereby realizing breakpoint recovery training.
[0024] The innovative aspects of this invention include, but are not limited to, the following: 1. In this embodiment, the preprocessed data pools of each configuration are equally divided into N fragments, and fragments with the same sequence number under different configuration data pools are assigned to the same node. In this way, each computing node does not obtain a certain segment of the global data, but a miniature subset of the global dataset containing all heterogeneous classifications. This is one of the innovations of this embodiment.
[0025] 2. In this embodiment, before each iteration begins, each computing node independently executes modal decision-making to ensure that each iteration processes only data of a single modality. In this way, forward interference and loss scale conflicts between different modalities can be physically isolated during the forward and backward propagation stages, which is one of the innovative points of this embodiment.
[0026] 3. In this embodiment, before each iteration begins, each computing node independently executes modal decision-making, ensuring that each computing node loads data of the same modality in a training round. Therefore, it can be ensured that the length difference of the token sequence within the same batch is small, avoiding the need for a large number of padding operations due to large differences in the sequence length of the data. Padding can be minimized, effectively reducing the padding waste caused by the mixing of long and short sequences, thereby saving GPU memory and forward / backward propagation computing power. This is one of the innovative points of this embodiment.
[0027] 4. In this embodiment, implicit data allocation is achieved through cluster-wide All-Reduce, enabling large models to smoothly balance visual language understanding and precise embodied action control, which is one of the innovative points of this embodiment.
[0028] 5. In this embodiment, GPU computation flow isolation and gradient fusion are performed within the same cluster according to the data modality, which reduces the space complexity from O(n) to O(n / N), where n is the total amount of data and N is the number of training nodes. Furthermore, as the number of training nodes increases, the memory space usage will be further reduced. This can fundamentally avoid the problem of memory crashes when reading massive amounts of long trajectory motion data of robots, which is one of the innovations of this embodiment. Attached Figure Description
[0029] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0030] Figure 1 A flowchart of a training method provided in an embodiment of the present invention; Figure 2 A flowchart of a preprocessing method provided in an embodiment of the present invention; Figure 3 This is a flowchart illustrating the preprocessing of a visual language understanding dataset according to an embodiment of the present invention. Figure 4This is a flowchart of a preprocessing method for a visual language action dataset provided in an embodiment of the present invention; Figure 5 A flowchart for a probabilistic decision-making method; Figure 6 Another flowchart of the training method provided in an embodiment of the present invention; Figure 7 A schematic diagram of a training system provided in an embodiment of the present invention; Figure 8 This is another schematic diagram of the training system provided in an embodiment of the present invention. Detailed Implementation
[0031] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0032] It should be noted that the terms "comprising" and "having," and any variations thereof, in the embodiments and drawings of this invention are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the steps or units listed, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.
[0033] This invention discloses a distributed training method and system for multimodal data. These will be described in detail below.
[0034] Figure 1 A flowchart of a training method provided in an embodiment of the present invention is provided below. Figure 1 The multimodal data distributed training method provided in this embodiment of the invention includes: Step 1: Preprocess the original multimodal data to obtain multiple configuration data pools.
[0035] Different modalities of data, such as visual language understanding data and visual language action data, have large differences in sequence length and inconsistent loss scales. Therefore, when training these data together, it is easy to cause problems such as wasted GPU memory and gradient disorder.
[0036] To address the aforementioned problems, this invention preprocesses the original multimodal data to completely separate the data from different modalities. Figure 2 A flowchart of a preprocessing method provided in an embodiment of the present invention is provided below. Figure 2The raw multimodal data is preprocessed, specifically including: Step 11: Classify the original multimodal data according to data type to obtain the visual language understanding dataset and the visual language action dataset.
[0037] Step 12: Preprocess the visual language understanding dataset and the visual language action dataset to obtain the visual language understanding configuration data pool and the visual language action configuration data pool, respectively.
[0038] Typically, visual language understanding data (VLM data) contains simple text-based question-and-answer sequences, while visual language action data (VLA data) contains high-frequency action sequences and images and text commands from multiple perspectives. It is evident that VLA data has more modalities (such as perception and action) than VLM data, and there is an inconsistency in the action space dimension between different robots. Therefore, when preprocessing visual language action datasets, it is necessary to address the problem of action space heterogeneity, a problem that does not exist for VLM data. Thus, different preprocessing methods are required for visual language understanding datasets and visual language action datasets.
[0039] Figure 3 This is a flowchart illustrating the preprocessing of a visual language understanding dataset according to an embodiment of the present invention. Please refer to it. Figure 3 Preprocessing of the visual language understanding dataset includes: Step 1211: Fill the modal data of each modality in the visual language understanding dataset into the preset visual language understanding template, and parse and segment the filled string to obtain the visual language understanding token sequence; wherein, the modal data in the visual language understanding dataset includes at least image data and text question answering data.
[0040] To simplify the overall logic, a visual language understanding template is pre-constructed in this embodiment. By filling the corresponding positions in the visual language understanding template with various modal data from the visual language understanding dataset, such as image data and text question-and-answer data, a string with a unified format can be obtained. By parsing and segmenting the unified format string, the tokens that the model can understand can be obtained, serving as a visual language understanding token sequence.
[0041] Step 1212: Unify the visual language understanding token sequence into a semantic ontology pool to obtain the visual language understanding configuration data pool.
[0042] Unlike VLA data, VLM data does not contain action terms, and therefore does not suffer from the problem of action space heterogeneity. Thus, in this embodiment, all visual language understanding token sequences are unified into a single semantic ontology pool, serving as a visual language understanding configuration data pool. Consequently, during subsequent sharding, a uniform sharding strategy can be directly applied to this visual language understanding configuration data pool to allocate VLM data, without the need for grouping by configuration.
[0043] Different robot hardware (such as dual-arm, single-arm, robotic arms with different degrees of freedom, humanoid, etc.) have completely different motion space dimensions. In order to eliminate configuration differences, this embodiment performs cross-ontology data training on the visual language action dataset in the preprocessing stage, maps the robot action data of different configurations into a unified action vector and inputs it into the model, so that the model can process the action data of different configurations in a unified way.
[0044] Figure 4 This is a flowchart illustrating the preprocessing of a visual language action dataset according to an embodiment of the present invention. Please refer to it. Figure 4 Preprocessing of the visual language action dataset includes: Step 1221: Define a standard virtual ontology. The standard virtual ontology includes multiple semantic blocks divided by physical components, and assigns a fixed maximum dimension to each semantic block.
[0045] To break free from the limitations of a single robot configuration, this embodiment defines a standard virtual ontology at the global level, encompassing all possible physical components. This standard virtual ontology divides the action space into several non-overlapping semantic blocks. Each semantic block corresponds to a type of physical component, and a fixed maximum dimension is assigned to each semantic block, as detailed below: The left arm is assigned m1 dimension (e.g., 7-dimensional joint angle); the right arm is assigned m2 dimension (e.g., 7-dimensional joint angle); the left end effector is assigned m3 dimension (e.g., 1-dimensional gripper opening and closing); the right end effector is assigned m4 dimension (e.g., 1-dimensional gripper opening and closing); the torso is assigned m5 dimension (e.g., 2-dimensional lifting and rotation); and the chassis is assigned m6 dimension (e.g., 2-dimensional linear velocity and angular velocity).
[0046] Step 1222: Decompose the original action vector according to the component type and fill it into the corresponding semantic block. Then, splice the filled semantic blocks in a predetermined physical order to form a unified action vector.
[0047] Once a standard virtual ontology is defined, the original action vector can be decomposed and filled in according to the correspondence between component types and semantic blocks in the standard virtual ontology.
[0048] It should be noted that during the padding process, if the dimension of the action data is less than the maximum dimension of the semantic block, padding is performed at the end of the semantic block, such as by adding zeros or a preset mask value, to reach the maximum dimension. If the current robot does not have the component, the corresponding semantic block is filled with a preset mask value. For example, if the current data is a pure single-arm fixed robot without a right arm and chassis, the semantic blocks corresponding to "right arm," "right end effector," and "chassis" are all filled with preset mask values, indicating that the current robot corresponding to that semantic block is an invalid component.
[0049] After infilling, the infilled semantic blocks are concatenated in a predetermined physical order to form a unified action vector. Thus, regardless of the robot data configuration input, the i-th dimension of the final vector will always control only a specific physical part (e.g., Dim 8 will always only control the shoulder yaw angle of the right arm). This allows the network weights of generative models such as stream matching to stably learn a deterministic mapping from visual / textual features to specific body part instructions, significantly improving the convergence quality of mixed training. Because the left and right hands, end effectors, and other parts are aligned in the feature space, the "fine-grasping strategy" learned by the model on dual-arm robot data can be directly transferred and used to control the corresponding robotic arm of a single-arm robot with zero-shot data.
[0050] Step 1223: Fill the modal data and unified action vectors in the visual language action dataset into the preset visual language action template; wherein, the modal data in the visual language action dataset includes at least image data and text instruction data.
[0051] Step 1224: Parse and segment the padded string to obtain the visual language action token sequence.
[0052] To simplify the overall logic, a visual language action template is pre-constructed in this embodiment. By filling the corresponding positions in the visual language understanding template with various modal data from the visual language action dataset, such as image data, text command data, and unified action vectors, a string with a unified format can be obtained. By parsing and segmenting the unified format string, the lexical units that the model can understand can be obtained, which serve as the visual language action token sequence.
[0053] Furthermore, the visual language understanding template and the visual language action template can also use a unified data template. In this embodiment, the following template is used uniformly: "{CHAT_USER_PREFIX}<image_image_!>{BOS}Embodiment: <embodiment_text_!>; Task:<command_text_!_200> State: <proprio_proprio_!>{CHAT_USER_SUFFIX}{CHAT_ASSISTANT_PREFIX}" <eov> <eoc>Action: <action_action>|{EOS}”. Among them, CHAT_USER_PREFIX represents the placeholder at the beginning of the question; image_image_! represents the image input; BOS represents the start symbol; Embodiment: embodiment_text_! represents the robot configuration; Task: <command_text_!_200> represents the original instruction of the robot action; State: <proprio_proprio_! represents the state of the robot body; CHAT_USER_SUFFIX represents the placeholder at the end of the question; CHAT_ASSISTANT_PREFIX represents the placeholder at the beginning of the answer; <eov> <eoc>This is the end marker; Action:<action_action> {EOS} represents robot actions; {EOS} represents the terminator.
[0054] By using a unified template for template processing, heterogeneous VLM and VLA data can be completely transformed into the same language sequence. Therefore, the cross-entropy loss function can be used uniformly for gradient backpropagation, which is simple to implement and can solve the problem of gradient superposition and conflict between two different optimization directions within a single card when the existing technology forcibly mixes calculations in the same batch. This makes it difficult for the optimizer to find the global optimum, resulting in catastrophic forgetting or decreased accuracy of action output. Thus, it solves the gradient disorder when training visual language understanding data and multi-dimensional action data together, and improves the efficiency of mixed training.
[0055] Step 1225: Based on the physical ontology type corresponding to the visual language action token sequence, divide the visual language action token sequence into multiple visual language action configuration data pools.
[0056] Robots with different physical ontology types have different motion space dimensions, resulting in different compositions of action lexical units in their corresponding visual language action token sequences. Based on this, this embodiment divides the visual language action token sequences into configuration pools according to physical ontology type. Visual language action token sequences belonging to the same physical ontology type are grouped into the same visual language action configuration data pool, ultimately generating multiple visual language action configuration data pools. Each visual language action configuration data pool is specifically used to store data belonging to the same robot configuration; for example, a configuration data pool for robot A-type robotic arm, a configuration data pool for robot B-type robotic arm, and a configuration data pool for dexterous hand-type robots, etc.
[0057] This invention uses a preprocessing step to completely separate different modal data (such as visual language understanding data and visual language action data of different configurations) into multiple data pools of different configurations, ensuring that the data in the same configuration data pool are all homogeneous batches, which facilitates the segmented loading of multimodal data according to configuration data pools.
[0058] Step 2: Divide each configuration data pool into N equal parts, and assign the parts with the same index in different configuration data pools to the same computing node; where N represents the total number of computing nodes currently participating in training.
[0059] After obtaining the data pools for each configuration, each configuration pool needs to be sharded. In this invention, based on the total number of computing nodes currently participating in training, each configuration data pool, divided during the preprocessing stage, is independently and proportionally sharded. For example, assuming there are N computing nodes, each configuration data pool is equally sharded into N shards. Then, the mapping scheduler uniformly allocates the i-th shard from different configuration data pools to the i-th node in the cluster, where i∈{1, 2, ...,N}.
[0060] Using the above allocation method, each computing node does not obtain a specific segment of the global data, but rather a miniature subset of the global dataset containing all heterogeneous classifications.
[0061] Step 3: Each computing node uses the assigned shard number as an index to load the corresponding shard data subset.
[0062] During the data instantiation phase, each compute node strictly uses its assigned shard number as an index to initialize its local sampler and loader. By configuring compute nodes to load only the subset of shard data corresponding to their own index (approximately 1 / N of the total global data), instead of reading the global dataset metadata, the space complexity is reduced from O(n) to O(n / N), where n is the total data volume and N is the number of training nodes. Furthermore, as the number of training compute nodes increases, memory usage further decreases, fundamentally avoiding memory crashes caused by reading massive amounts of long-trajectory robot motion data.
[0063] Step 4: Before the start of each iteration, each computing node independently performs modal decision-making to obtain the modal decision-making results of this round of training.
[0064] To address the problem of gradient disorder caused by forcibly mixing data of different modalities within the same batch in existing technologies, this invention proposes that each computing node independently executes modal decisions before each iteration, ensuring that each iteration processes only data of a single modality. Thus, during the forward and backward propagation phases, since each computing node processes only one modality of data, forward interference and loss scale conflicts between different modalities are physically isolated. This avoids the superposition and conflict of gradients from different optimization directions within a single card, preventing the optimizer from finding the global optimum and causing problems such as "catastrophic forgetting" or decreased action output accuracy.
[0065] In this invention, different methods can be used to perform modal decision-making, such as probabilistic decision-making methods or deterministic decision-making methods. The specific method can be selected according to the actual scenario, and this application does not impose any specific limitations on it.
[0066] Figure 5 Here is a flowchart of a probabilistic decision-making method; please refer to it. Figure 5 Modal decision-making is performed using probabilistic decision-making methods, specifically including: Step 41: Set the sampling probability of visual language understanding data to p, then the sampling probability of visual language action data is 1-p.
[0067] The data types include visual language understanding data (VLM data) and visual language action data (VLA data). The sampling probability of VLM data is set to p, and p∈[0,1]. Then the sampling probability of VLA data is 1-p. The sampling probabilities of the two are used to control the ratio of VLM data and VLA data during the training process.
[0068] Step 42: Each computing node generates a random number r, r∈[0,1], based on a uniform distribution.
[0069] Step 43: Compare r and p. If r < p, the corresponding computing node will perform the visual language understanding task in the current iteration step; otherwise, the corresponding computing node will perform the visual language action task in the current iteration step.
[0070] Step 5: Based on the modality decision results, select the corresponding data loader to load the homogeneous batch training data of the corresponding modality from the data subset of that node.
[0071] When the modality decision result indicates that the current training modality is VLM data, the VLM loader loads the visual language understanding token sequence corresponding to the shard index from the visual language understanding configuration data pool according to the shard index corresponding to the current node.
[0072] When the modal decision result indicates that the current training modality is VLA data and the configuration type is robotic arm type A, the VLA loader loads the visual language action token sequence corresponding to the corresponding shard index from the corresponding visual language action configuration data pool (such as the robotic arm type A configuration data pool) according to the shard index corresponding to the current node.
[0073] Since each computing node loads data of the same modality in a training round, it can ensure that the length difference of the token sequence within the same batch is small. This avoids the need for a large number of padding operations due to large differences in the sequence length of the data. Padding can be minimized, effectively reducing the padding waste caused by the mixing of long and short sequences, thereby saving GPU memory and forward / backward propagation computing power.
[0074] Step 6: Each computing node performs forward and backward propagation on the homogeneous batch training data in sequence to obtain the gradient of each node.
[0075] VLM data contains both image and text data. Therefore, when performing forward propagation on the VLM training data, it is necessary to extract image and text features from the image and text data in the VLM data and then fuse them. The fused multimodal features are then input into the VLM model for forward propagation, and the first loss is calculated. Here, the first loss refers to the cross-entropy loss, denoted as... Where n represents the total number of categories; Let be the logits value for the i-th class, representing the score of the i-th class output by the network. Let be the logits value for the j-th class, representing the score for the j-th class output by the network. This represents the true label of the i-th class.
[0076] VLA data includes not only image and text data but also action data. Therefore, when performing forward propagation on VLA training data, image features and text features need to be extracted separately and input into the VLA model together with a unified action vector for forward propagation to output predicted actions. At the same time, a second loss is calculated by comparing the predicted actions with the real labels.
[0077] The second loss consists of discrete action loss and continuous action loss. Discrete actions use the cross-entropy loss function, while continuous actions use the flow matching loss function. The cross-entropy loss function can be found above and will not be repeated here.
[0078] The flow matching loss function is expressed as follows: ,in, Indicates a time index. ∈[0,1], corresponding to the initial action distribution ( =0) Distribution of action toward the target ( The interpolation process of =1); This represents the learnable parameters of the model; This indicates the intermediate interpolation action state at time t; Indicates time index The current action state; Indicates conditional observation; Represents a conditional distribution, i.e., given observations Marginal distribution of the next action; Represents a simple linear Gaussian distribution; This represents the velocity field predicted by the model, and the output is a vector with the same dimension as the target velocity field; Represents the true velocity field; This represents the joint expectation of the time step, conditional distribution, and marginal distribution.
[0079] Based on the calculated first or second loss, each computing node executes the backpropagation algorithm to obtain the gradient of the corresponding computing node.
[0080] During the forward and backward propagation phases, since each computation node only processes one mode, the loss and gradient calculated internally are singular, which physically isolates the forward interference and loss scale conflict between VLM and VLA.
[0081] Step 7: Perform global gradient fusion on the gradients of all computing nodes to obtain the global gradient.
[0082] After backpropagation, gradient synchronization is performed using an All-Reduce operation, and the cluster globally aggregates and averages the gradients from all computing nodes. Due to the law of large numbers, in a cluster of tens to hundreds of nodes, each node independently selects VLM or VLA based on the sampling probability. Thus, when the number of nodes is large enough, the composition of the global gradient will naturally approach a weighted fusion of p's VLM gradient and 1-p's VLA gradient, achieving the expected data matching at the global update level.
[0083] Step 8: Update the model parameters using global gradients on all computing nodes.
[0084] The multimodal data distributed training method provided by this invention allows each computing node to independently execute modal decisions before each iteration, ensuring that each iteration processes only data of a single modality. This not only physically isolates forward interference and loss scale conflicts between different modalities during the forward and backward propagation stages, but also ensures that each computing node loads data of the same modality in each training round. Therefore, it ensures that the length difference of the token sequences within the same batch is small, avoiding the need for extensive padding operations due to large differences in data sequence lengths. This minimizes padding waste caused by mixing long and short sequences, thereby saving GPU memory and forward / backward propagation computing power.
[0085] Furthermore, implicit data allocation is achieved through cluster-wide All-Reduce, enabling large models to smoothly balance visual language understanding with precise embodied action control. Within the same cluster, GPU computation flow isolation and gradient fusion are performed based on data modalities, reducing space complexity from O(n) to O(n / N), where n is the total amount of data and N is the number of training nodes. Moreover, memory usage further decreases as the number of training nodes increases.
[0086] Optionally, Figure 6 For another flowchart of the training method provided in this embodiment of the invention, please refer to... Figure 6 Based on the modality decision results, the corresponding data loader is selected to load homogeneous batch training data of the corresponding modality from the data subset of that node, which also includes: Step 51: When training is interrupted, each data loader creates an iterator based on the interruption position. The iterator performs index jumps based on the fragment number and inner offset to locate the interruption position and continue reading data, thus realizing breakpoint recovery training.
[0087] To avoid the problem of GPU computing power being idle due to the need to iterate the loader from scratch after a restart caused by training interruption, this embodiment designs a large-scale shard sampling algorithm that supports breakpoint recovery. During system operation, the current shard number and internal offset of each data loader are recorded in real time. Thus, when training is interrupted, regardless of how many millions of steps were trained before the interruption, the read point can be directly located by using iterator sharding for index jumping after restarting. Therefore, the mechanism of re-traversing the dataset can be avoided, bypassing actual disk reads and CPU preprocessing, compressing the recovery time of traditional hours to within seconds, effectively reducing time complexity, and thus greatly improving the effective computing time of the GPU cluster.
[0088] Based on the same inventive concept, this invention also provides a multimodal data distributed training system. Figure 7 Please refer to the schematic diagram of a training system provided in an embodiment of the present invention. Figure 7 The multimodal data distributed training system 100 provided in this embodiment of the invention includes: Preprocessing module 110 is used to preprocess the raw multimodal data to obtain multiple configuration data pools; The partitioning module 120 is used to equally partition each configuration data pool into N partitions and assign partitions with the same index in different configuration data pools to the same computing node; where N represents the total number of computing nodes currently participating in training; The loading module 130 is used to enable each computing node to load the corresponding data subset of the shard based on the allocated shard number as an index. The decision module 140 is used to enable each computing node to independently execute modal decisions before the start of each iteration, so as to obtain the modal decision results of this round of training. The filtering module 150 is used to select the corresponding data loader from the data subset of the node to load the homogeneous batch training data of the corresponding mode based on the modality decision result. The execution module 160 is used to enable each computing node to perform forward propagation and backward propagation on the homogeneous batch training data in sequence to obtain the gradient of each node; The fusion module 170 is used to perform global gradient fusion on the gradients of all computing nodes to obtain the global gradient; Update module 180 is used to enable all computing nodes to update model parameters using global gradients.
[0089] The above system embodiments correspond to the method embodiments and have the same technical effects as the method embodiments. For details, please refer to the method embodiments, which will not be repeated here.
[0090] Optionally, the preprocessing module 110 is specifically configured as follows: The original multimodal data were classified according to data type to obtain the visual language understanding dataset and the visual language action dataset. The visual language understanding dataset and the visual language action dataset are preprocessed separately to obtain the visual language understanding configuration data pool and the visual language action configuration data pool.
[0091] Optionally, when preprocessing the visual language understanding dataset, the preprocessing module 110 is specifically configured as follows: The modal data from the visual language understanding dataset are populated into a preset visual language understanding template, and the populated strings are parsed and segmented to obtain a visual language understanding token sequence; wherein, the modal data in the visual language understanding dataset includes at least image data and text question-answering data; By unifying the visual language understanding token sequence into a semantic ontology pool, a visual language understanding configuration data pool is obtained.
[0092] Optionally, when preprocessing the visual language action dataset, the preprocessing module 110 is specifically configured as follows: Define a standard virtual ontology, which consists of multiple semantic blocks divided by physical components, and assign a fixed maximum dimension to each semantic block; The original action vector is decomposed according to component type and filled into the corresponding semantic blocks. The filled semantic blocks are then spliced together in a predetermined physical order to form a unified action vector. The modal data and unified action vectors from the visual language action dataset are populated into a preset visual language action template; wherein, the modal data from the visual language action dataset includes at least image data and text instruction data; The padded string is parsed and segmented to obtain a visual language action token sequence; Based on the physical ontology type corresponding to the visual language action token sequence, the visual language action token sequence is divided into multiple visual language action configuration data pools.
[0093] Optionally, the decision module 140 is specifically configured as follows: Modal decision-making can be performed using probabilistic decision-making methods or deterministic decision-making methods.
[0094] Optionally, the data types include visual language understanding data and visual language action data; when a probabilistic decision-making method is used to perform modal decision-making, the decision module 140 is specifically configured as follows: If the sampling probability of visual language understanding data is set to p, then the sampling probability of visual language action data is 1-p; Each computing node generates a random number r, r∈[0,1], based on a uniform distribution. Compare r and p. If r < p, the corresponding computing node will perform the visual language understanding task in the current iteration step; otherwise, the corresponding computing node will perform the visual language action task in the current iteration step.
[0095] Optionally, Figure 8 For another structural schematic diagram of the training system provided in this embodiment of the invention, please refer to... Figure 8 In this embodiment, the multimodal data distributed training system 100 also includes a positioning module 190. The positioning module 190 is used to enable each data loader to create an iterator according to the interruption position when training is interrupted. The iterator performs index jump according to the fragment number and the inner offset to locate the interruption position and continue reading data, thereby realizing breakpoint recovery training.
[0096] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.
[0097] Those skilled in the art will understand that the modules in the apparatus of the embodiments can be distributed in the apparatus of the embodiments as described in the embodiments, or they can be located in one or more devices different from this embodiment with corresponding changes. The modules of the above embodiments can be combined into one module, or they can be further divided into multiple sub-modules.
[0098] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.< / eoc> < / eov> < / eoc> < / eov>
Claims
1. A distributed training method for multimodal data, characterized in that, include: The original multimodal data is preprocessed to obtain multiple configuration data pools; Each configuration data pool is equally divided into N fragments, and fragments with the same index in different configuration data pools are assigned to the same computing node; where N represents the total number of computing nodes currently participating in training; Each computing node uses its assigned shard number as an index to load the corresponding shard data subset. Before each iteration begins, each computing node independently performs modal decision-making to obtain the modal decision-making results for this round of training; Based on the modality decision result, select the corresponding data loader to load the homogeneous batch training data of the corresponding modality from the data subset of that node; Each computing node sequentially performs forward and backward propagation on the homogeneous batch training data to obtain the gradient of each node; Global gradient fusion is performed on the gradients of all computing nodes to obtain the global gradient; All computing nodes use the global gradient to update model parameters.
2. The multimodal data distributed training method according to claim 1, characterized in that, The raw multimodal data is preprocessed, specifically including: The original multimodal data were classified according to data type to obtain the visual language understanding dataset and the visual language action dataset. The visual language understanding dataset and the visual language action dataset are preprocessed to obtain a visual language understanding configuration data pool and a visual language action configuration data pool, respectively.
3. The multimodal data distributed training method according to claim 2, characterized in that, Preprocessing of the visual language understanding dataset specifically includes: The modal data in the visual language understanding dataset are filled into a preset visual language understanding template, and the filled strings are parsed and segmented to obtain a visual language understanding token sequence; wherein, the modal data in the visual language understanding dataset includes at least image data and text question-answering data; The visual language understanding token sequence is unified into a semantic ontology pool to obtain a visual language understanding configuration data pool.
4. The multimodal data distributed training method according to claim 2, characterized in that, Preprocessing the visual language action dataset specifically includes: Define a standard virtual ontology, which includes multiple semantic blocks divided by physical components, and assign a fixed maximum dimension to each semantic block; The original action vector is decomposed according to component type and filled into the corresponding semantic blocks. The filled semantic blocks are then spliced together in a predetermined physical order to form a unified action vector. The modal data and unified action vectors in the visual language action dataset are filled into a preset visual language action template; wherein, the modal data in the visual language action dataset includes at least image data and text instruction data; The padded string is parsed and segmented to obtain a visual language action token sequence; Based on the physical ontology type corresponding to the visual language action token sequence, the visual language action token sequence is divided into multiple visual language action configuration data pools.
5. The multimodal data distributed training method according to claim 4, characterized in that, When the original motion vector is decomposed according to component type and filled into the corresponding semantic block, if the current robot has the component, the motion data is filled into the corresponding semantic block. If the dimension is less than the maximum dimension of the semantic block, the data is only filled at the end of the semantic block. If the current robot does not have the component, the corresponding semantic block is filled with a preset mask value.
6. The multimodal data distributed training method according to claim 2, characterized in that, Performing modal decisions specifically includes: Modal decision-making can be performed using probabilistic decision-making methods or deterministic decision-making methods.
7. The multimodal data distributed training method according to claim 6, characterized in that, The data types include visual language understanding data and visual language action data; Modal decision-making is performed using probabilistic decision-making methods, specifically including: If the sampling probability of visual language understanding data is set to p, then the sampling probability of visual language action data is 1-p; Each computing node generates a random number r, r∈[0,1], based on a uniform distribution. Compare r and p. If r < p, the corresponding computing node will perform the visual language understanding task in the current iteration step; otherwise, the corresponding computing node will perform the visual language action task in the current iteration step.
8. The multimodal data distributed training method according to claim 1, characterized in that, Based on the modality decision result, selecting the corresponding data loader to load homogeneous batch training data of the corresponding modality from the data subset of that node, further includes: When training is interrupted, each data loader creates an iterator based on the interruption location. The iterator performs index jumps based on the fragment number and inner offset to locate the interruption location and continue reading data, thus realizing breakpoint resumption of training.
9. A multimodal data distributed training system, characterized in that, include: The preprocessing module is used to preprocess the raw multimodal data to obtain multiple configuration data pools; The partitioning module is used to equally partition each configuration data pool into N partitions and assign partitions with the same index in different configuration data pools to the same computing node; where N represents the total number of computing nodes currently participating in training; The loading module is used to enable each computing node to load the corresponding data subset of the shard based on the allocated shard number as an index. The decision module is used to enable each computing node to independently execute modal decisions before the start of each iteration, and obtain the modal decision results of this round of training; The filtering module is used to select the corresponding data loader from the data subset of the node to load the homogeneous batch training data of the corresponding mode based on the modality decision result. The execution module is used to enable each computing node to sequentially perform forward and backward propagation on the homogeneous batch training data to obtain the gradient of each node; The fusion module is used to perform global gradient fusion on the gradients of all computing nodes to obtain the global gradient; The update module is used to enable all computing nodes to update model parameters using the global gradient.
10. The multimodal data distributed training system according to claim 9, characterized in that, The preprocessing module is specifically used for: The original multimodal data were classified according to data type to obtain the visual language understanding dataset and the visual language action dataset. The visual language understanding dataset and the visual language action dataset are preprocessed to obtain a visual language understanding configuration data pool and a visual language action configuration data pool, respectively.
11. The multimodal data distributed training system according to claim 10, characterized in that, When preprocessing the visual language understanding dataset, the preprocessing module is specifically used for: The modal data in the visual language understanding dataset are filled into a preset visual language understanding template, and the filled strings are parsed and segmented to obtain a visual language understanding token sequence; wherein, the modal data in the visual language understanding dataset includes at least image data and text question-answering data; The visual language understanding token sequence is unified into a semantic ontology pool to obtain a visual language understanding configuration data pool.
12. The multimodal data distributed training system according to claim 10, characterized in that, When preprocessing the visual language action dataset, the preprocessing module is specifically used for: Define a standard virtual ontology, which includes multiple semantic blocks divided by physical components, and assign a fixed maximum dimension to each semantic block; The original action vector is decomposed according to component type and filled into the corresponding semantic blocks. The filled semantic blocks are then spliced together in a predetermined physical order to form a unified action vector. The modal data and unified action vectors in the visual language action dataset are filled into a preset visual language action template; wherein, the modal data in the visual language action dataset includes at least image data and text instruction data; The padded string is parsed and segmented to obtain a visual language action token sequence; Based on the physical ontology type corresponding to the visual language action token sequence, the visual language action token sequence is divided into multiple visual language action configuration data pools.
13. The multimodal data distributed training system according to claim 10, characterized in that, The decision-making module is specifically used for: Modal decision-making can be performed using probabilistic decision-making methods or deterministic decision-making methods.
14. The multimodal data distributed training system according to claim 13, characterized in that, The data types include visual language understanding data and visual language action data; when a probabilistic decision-making method is used to perform modal decision-making, the decision module is specifically used for: If the sampling probability of visual language understanding data is set to p, then the sampling probability of visual language action data is 1-p; Each computing node generates a random number r, r∈[0,1], based on a uniform distribution. Compare r and p. When r < p, the corresponding computing node performs the visual language understanding task in the current iteration step. Conversely, the corresponding computing node performs the visual language action task in the current iteration step.
15. The multimodal data distributed training system according to claim 9, characterized in that, It also includes a positioning module, which is used to enable each data loader to create an iterator based on the interruption position when training is interrupted. The iterator performs index jump based on the fragment number and inner offset to locate the interruption position and continue reading data, thereby realizing breakpoint recovery training.