Multi-modal processing method, device and equipment and readable storage medium

Abstract

The invention discloses a multi-modal processing method, device and equipment and a readable storage medium, and belongs to the technical field of machine learning. The method comprises the following steps: acquiring to-be-processed first data; multi-modal processing is carried out on the first data through a first model, a multi-modal processing result is obtained, and the first model is obtained through training of third training data and fourth training data; the third training data and the fourth training data are obtained by adjusting a first reference difference between the loaded first training data and second training data, and the first reference difference indicates the length discrete condition of the first training data and the second training data; a second reference difference between the third training data and the fourth training data is less than a first reference difference between the first training data and the second training data. By reducing the length discrete condition of the training data used for model training, the efficiency of obtaining the first model by training the training data with smaller length discrete condition is higher.

Description

Technical Field

[0001] This application relates to the field of machine learning technology, and in particular to multimodal processing methods, apparatus, devices and readable storage media. Background Technology

[0002] In the field of machine learning technology, a multimodal model is a machine learning model capable of processing various types of data in multiple modalities, including text, images, audio, and video. Multimodal processing using multimodal models includes, but is not limited to, natural language processing, computer vision, and speech recognition. Summary of the Invention

[0003] This application provides a multimodal processing method, apparatus, device, and readable storage medium to perform multimodal processing using a first model. The technical solution is as follows:

[0004] In a first aspect, a multimodal processing method is provided, comprising: acquiring first data to be processed; performing multimodal processing on the first data using a first model to obtain a multimodal processing result, wherein the first model is trained using third training data and fourth training data; wherein the third training data and fourth training data are obtained by adjusting a first reference difference between the loaded first training data and second training data, the first reference difference indicating the length dispersion of the first training data and the second training data, and a second reference difference between the third training data and the fourth training data being less than the first reference difference between the first training data and the second training data.

[0005] The adjustment of the first reference difference between the training data (first training data and second training data) is performed after the training data is loaded. That is, the first model of this application is obtained by adjusting the length dispersion online during training. The adjusted training data is the loaded training data, not all the training data in the training dataset. The amount of training data adjusted is small, resulting in high balancing efficiency. By reducing the first reference difference between the first and second training data, the length dispersion of the training data used for model training is reduced, making model training with training data with smaller length dispersion more efficient.

[0006] In one possible implementation, the first reference difference between the first training data and the second training data includes at least one of the following: the first reference difference is the reference difference between the data length of the first training data and the data length of the second training data; or, the first reference difference is the reference difference between the first length corresponding to the first training data and the first length corresponding to the second training data, wherein the first length corresponding to the first training data indicates the length of non-text data in the first training data; or, the first length corresponding to the first training data indicates the length of non-text data in the first training data and the fifth training data, and the first length corresponding to the second training data indicates the length of non-text data in the second training data; or, the first length corresponding to the second training data indicates the length of non-text data in the second training data and the sixth training data. This application does not limit the adjustment of the first reference difference; it can be any one or more of the first reference difference between data lengths and the first reference difference between first lengths, offering high flexibility.

[0007] In one possible implementation, the third training data is obtained based on the concatenated first training data, and the fourth training data is obtained based on the concatenated second training data. The concatenated first training data is obtained based on the fifth and first training data, and the concatenated second training data is obtained based on the sixth and second training data. The length of the concatenated first training data is equal to the sum of the lengths of the fifth and first training data, and the length of the concatenated second training data is equal to the sum of the lengths of the sixth and second training data. The second reference difference between the lengths of the concatenated first and second training data is less than the first reference difference. Here, the first reference difference indicates the length dispersion, which can be reduced by concatenating the training data. The adjustment process is simple and has low operational complexity.

[0008] In one possible implementation, the fifth training data is obtained from a selected data pile based on the type of the first training data. The multiple data piles are partitioned based on the type of the training data, and each data pile includes at least one loaded training data set. By obtaining the fifth training data based on the type of the first training data and ensuring that the first training data does not conflict with any other type, the acquisition accuracy of the fifth training data is high.

[0009] In one possible implementation, the third training data and the first training data correspond to the same data parallelism (DP) node, and the fourth training data and the second training data correspond to the same DP node. The first length corresponding to the first training data is less than the first length corresponding to the second training data. The fourth training data is obtained by moving selected non-text data from the non-text data on the DP node corresponding to the second training data, and the third training data is obtained by moving selected non-text data to the DP node corresponding to the first training data. The second reference difference between the first lengths corresponding to the fourth training data and the first lengths corresponding to the third training data is less than the first reference difference. The reduction of the first reference difference between the first lengths can be achieved simply by moving non-text data between DP nodes; the adjustment process is simple and efficient.

[0010] In one possible implementation, training may be interrupted during the process of training the first model. The first model can then be trained again after the interruption, based on training pointers or identifiers. These training pointers or identifiers are saved based on the number of iterations during model training and are used to identify unused training data. By saving the model's training pointers or identifiers, even if training is interrupted, training can continue using unused training data, ensuring reliable continuation of training from breakpoints.

[0011] In one possible implementation, model training is performed based on at least one data process and at least one DP node. If at least one of the numbers of DP nodes or data processes changes, the unused training data is adjusted based on at least one of the changed numbers of nodes or processes, within at least one dimension of the DP node or data process. Even if the DP node or data process changes, the training data indicated by the training pointer can be dynamically adjusted to avoid affecting the order in which subsequent training data is read, thus ensuring the normal execution of model training. This approach is applicable to scenarios where any DP node or data process changes, and has high versatility.

[0012] In one possible implementation, the first reference difference includes the reference difference between the lengths of the first and second training data. The first model is obtained through iterative training using the third and fourth training data, based on the total number of iterations. The total number of iterations indicates the number of times the first model was trained. Since the third and fourth training data are obtained by adjusting the first reference difference between their lengths (i.e., the lengths of the third and fourth training data differ from those of the first and second training data), by re-obtaining the total number of iterations, overfitting caused by too many or too few iterations is avoided, thus ensuring the equivalence of hyperparameters during training.

[0013] Secondly, a model training method is provided, comprising: acquiring loaded first training data and second training data; adjusting a first reference difference between the first training data and the second training data to obtain third training data and fourth training data, wherein the first reference difference indicates the length dispersion of the first training data and the second training data, and a second reference difference between the third training data and the fourth training data is less than the first reference difference between the first training data and the second training data; and training a first model based on the third training data and the fourth training data, wherein the first model is used for multimodal processing.

[0014] In one possible implementation, adjusting the first reference difference between the first training data and the second training data includes at least one of the following: adjusting the first reference difference between the data length of the first training data and the data length of the second training data; or, adjusting the first reference difference between the first length corresponding to the first training data and the first length corresponding to the second training data, wherein the first length corresponding to the first training data indicates the length of non-text data in the first training data, or, the first length corresponding to the first training data indicates the length of non-text data in the first training data and the fifth training data, wherein the first length corresponding to the second training data indicates the length of non-text data in the second training data, or, the first length corresponding to the second training data indicates the length of non-text data in the second training data and the sixth training data.

[0015] In one possible implementation, adjusting the first reference difference between the data length of the first training data and the data length of the second training data includes: acquiring fifth training data and sixth training data; concatenating the fifth training data and the first training data, concatenating the sixth training data and the second training data, wherein the second reference difference between the data length of the concatenated first training data and the data length of the concatenated second training data is less than the first reference difference.

[0016] In one possible implementation, obtaining the fifth training data includes: obtaining multiple data piles, each data pile including at least one training data, the multiple data piles being divided based on the type of training data; selecting a data pile from the multiple data piles based on the type of the first training data; and obtaining the fifth training data from the selected data pile.

[0017] In one possible implementation, the first length corresponding to the first training data is less than the first length corresponding to the second training data, the first training data and the third training data correspond to the same DP node, and the second training data and the fourth training data correspond to the same DP node; adjusting the first reference difference between the first length corresponding to the first training data and the first length corresponding to the second training data includes: selecting non-text type data from the non-text type data on the DP node corresponding to the second training data; moving the selected non-text type data to the DP node corresponding to the first training data to obtain the third training data and the fourth training data, wherein the second reference difference between the first length corresponding to the fourth training data and the first length corresponding to the third training data is less than the first reference difference.

[0018] In one possible implementation, the method further includes: if the number of iteration training steps at the current moment reaches the number of times to save, saving the training pointer at the current moment, the training pointer is used to identify unused training data at the current moment, and the training pointer is used to continue training after the model training is interrupted; or, if the number of iteration training steps at the current moment reaches the number of times to save, saving the identification information at the current moment, the identification information is used to identify unused training data at the current moment, and the identification information is used to continue training after the model training is interrupted.

[0019] In one possible implementation, model training is performed based on at least one data process and at least one DP node. After saving the training pointer at the current moment, the method further includes: if at least one of the number of DP nodes or the number of data processes changes, adjusting the unused training data in at least one dimension of the DP nodes or data processes based on the changed number of nodes or processes, and continuing training using the adjusted training data.

[0020] In one possible implementation, the third and fourth training data are obtained by adjusting the first reference difference between the data lengths of the first and second training data. Before training the first model based on the third and fourth training data, the method further includes: obtaining the total number of iterations corresponding to the third and fourth training data, where the total number of iterations indicates the number of times the first model has been iterated; training the first model based on the third and fourth training data includes: iteratively training the first model using the third and fourth training data based on the total number of iterations.

[0021] In one possible implementation, after training the first model based on the third and fourth training data, the method further includes: obtaining the first data of the first model; obtaining the multimodal processing result of the first data through the first model; and outputting the second data based on the multimodal processing result.

[0022] Thirdly, a multimodal processing apparatus is provided, comprising: an acquisition module for acquiring first data to be processed; and a processing module for performing multimodal processing on the first data using a first model to obtain a multimodal processing result, wherein the first model is trained using third training data and fourth training data; wherein the third training data and fourth training data are obtained by adjusting a first reference difference between the loaded first training data and second training data, the first reference difference indicating the length dispersion of the first training data and the second training data, and a second reference difference between the third training data and the fourth training data being less than the first reference difference between the first training data and the second training data.

[0023] In one possible implementation, the first reference difference between the first training data and the second training data includes at least one of the following: the first reference difference is the reference difference between the data length of the first training data and the data length of the second training data; or, the first reference difference is the reference difference between the first length corresponding to the first training data and the first length corresponding to the second training data, wherein the first length corresponding to the first training data indicates the length of non-text type data in the first training data; or, the first length corresponding to the first training data indicates the length of non-text type data in the first training data and the fifth training data, wherein the first length corresponding to the second training data indicates the length of non-text type data in the second training data; or, the first length corresponding to the second training data indicates the length of non-text type data in the second training data and the sixth training data.

[0024] In one possible implementation, the third training data is obtained based on the concatenated first training data, and the fourth training data is obtained based on the concatenated second training data; the concatenated first training data is obtained based on the fifth training data and the first training data, and the concatenated second training data is obtained based on the sixth training data and the second training data; the data length of the concatenated first training data is equal to the sum of the data lengths of the fifth training data and the first training data, and the data length of the concatenated second training data is equal to the sum of the data lengths of the sixth training data and the second training data; the second reference difference between the data lengths of the concatenated first training data and the data lengths of the concatenated second training data is less than the first reference difference.

[0025] In one possible implementation, the fifth training data is obtained from a selected data pile based on the type of the first training data. The multiple data piles are divided based on the type of training data, and any data pile includes at least one loaded training data.

[0026] In one possible implementation, the third training data and the first training data correspond to the same DP node, the fourth training data and the second training data correspond to the same DP node, the first length corresponding to the first training data is less than the first length corresponding to the second training data, the fourth training data is obtained by moving selected non-text data from the non-text data on the DP node corresponding to the second training data, the third training data is obtained by moving the selected non-text data to the DP node corresponding to the first training data, and the second reference difference between the first length corresponding to the fourth training data and the first length corresponding to the third training data is less than the first reference difference.

[0027] In one possible implementation, there is a training interruption during the training of the first model. The first model is then trained again based on the training pointer or identification information. The training pointer or identification information is saved based on the number of iterative training steps during the model training process. The training pointer or identification information is used to identify unused training data.

[0028] In one possible implementation, model training is performed based on at least one data process and at least one DP node. If at least one of the number of DP nodes or the number of data processes changes, the unused training data is adjusted based on at least one of the changed number of nodes or processes in at least one dimension of the DP nodes or data processes.

[0029] In one possible implementation, the first reference difference includes the reference difference between the data length of the first training data and the data length of the second training data. The first model is obtained by iterative training using the third and fourth training data based on the total number of iterations. The total number of iterations indicates the number of times the first model was obtained through iterative training.

[0030] Fourthly, a model training apparatus is provided, comprising: an acquisition module for acquiring loaded first training data and second training data; an adjustment module for adjusting a first reference difference between the first training data and the second training data to obtain third training data and fourth training data, wherein the first reference difference indicates the length dispersion of the first training data and the second training data, and a second reference difference between the third training data and the fourth training data is less than the first reference difference between the first training data and the second training data; and a training module for training a first model based on the third training data and the fourth training data, wherein the first model is used for multimodal processing.

[0031] In one possible implementation, the adjustment module is configured to perform at least one of the following: adjusting a first reference difference between the data volume length of the first training data and the data length of the second training data; or, adjusting a first reference difference between the first length corresponding to the first training data and the first length corresponding to the second training data, wherein the first length corresponding to the first training data indicates the length of non-text type data in the first training data; or, the first length corresponding to the first training data indicates the length of non-text type data in the first training data and the fifth training data, wherein the first length corresponding to the second training data indicates the length of non-text type data in the second training data; or, the first length corresponding to the second training data indicates the length of non-text type data in the second training data and the sixth training data.

[0032] In one possible implementation, an adjustment module is used to acquire fifth training data and sixth training data; concatenate the fifth training data and the first training data, and concatenate the sixth training data and the second training data, wherein the second reference difference between the data length of the concatenated first training data and the data length of the concatenated second training data is less than the first reference difference.

[0033] In one possible implementation, an adjustment module is used to acquire multiple data piles, each data pile including at least one training data, the multiple data piles being divided based on the type of the training data; a data pile is selected from the multiple data piles based on the type of the first training data, and a fifth training data is acquired from the selected data pile.

[0034] In one possible implementation, the third training data and the first training data correspond to the same DP node, the fourth training data and the second training data correspond to the same DP node, and the first length corresponding to the first training data is less than the first length corresponding to the second training data; the adjustment module is used to select non-text type data from the non-text type data on the DP node corresponding to the second training data; and move the selected non-text type data to the DP node corresponding to the first training data to obtain the third training data and the fourth training data, wherein the second reference difference between the first length corresponding to the fourth training data and the first length corresponding to the third training data is less than the first reference difference.

[0035] In one possible implementation, the training module is further configured to save the training pointer at the current moment when the number of iterations in the current moment reaches the save count. The training pointer is used to identify unused training data at the current moment and is used to continue training after model training is interrupted. Alternatively, when the number of iterations in the current moment reaches the save count, the module may save the identification information at the current moment. The identification information is used to identify unused training data at the current moment and is used to continue training after model training is interrupted.

[0036] In one possible implementation, model training is performed based on at least one data process and at least one DP node. The training module is further configured to adjust unused training data in at least one dimension of the DP node or data process based on the changed number of nodes or processes, and continue training using the adjusted training data, should at least one of the number of DP nodes or data processes change.

[0037] In one possible implementation, the third and fourth training data are obtained by adjusting the first reference difference between the data lengths of the first and second training data. The adjustment module is also used to obtain the total number of iteration steps corresponding to the third and fourth training data, whereby the total number of iteration steps indicates the number of times the first model is obtained through iterative training. The training module is used to iteratively train the first model using the third and fourth training data based on the total number of iteration steps.

[0038] In one possible implementation, the acquisition module is further configured to acquire first data of the first model; the apparatus further includes: a processing module configured to acquire multimodal processing results of the first data through the first model, and output second data based on the multimodal processing results.

[0039] Fifthly, a computer device is provided, the device including a processor for loading and executing at least one instruction to cause the computer device to perform the method of the first aspect or any possible implementation of the first aspect, or to perform the method of the second aspect or any possible implementation of the second aspect.

[0040] In one possible implementation, the device includes a memory coupled to a processor, the memory storing at least one instruction.

[0041] In a sixth aspect, a computer-readable storage medium is provided, wherein at least one instruction is stored in the computer-readable storage medium, the instruction being loaded and executed by a processor to implement the method of the first aspect or any possible implementation thereof, or to implement the method of the second aspect or any possible implementation thereof.

[0042] In a seventh aspect, a computer program (product) is provided, comprising computer program / instructions, which are executed by a processor to cause a computer to implement the method of the first aspect or any possible implementation of the first aspect, or to implement the method of the second aspect or any possible implementation of the second aspect.

[0043] Eighthly, a communication device is provided, comprising: a transceiver, a memory, and a processor. The transceiver, the memory, and the processor communicate with each other via an internal connection path. The memory stores instructions, and the processor executes the instructions stored in the memory to control the transceiver to receive and transmit signals. When the processor executes the instructions stored in the memory, it causes the processor to perform the method of the first aspect or any possible implementation of the first aspect, or to perform the method of the second aspect or any possible implementation of the second aspect. Optionally, the communication device may be a chip.

[0044] Optionally, there may be one or more processors and one or more memories.

[0045] Alternatively, the memory can be integrated with the processor, or the memory can be set up separately from the processor.

[0046] In the specific implementation process, the memory can be a non-transitory memory, such as read-only memory (ROM), which can be integrated with the processor on the same chip or set on different chips. This application does not limit the type of memory or the way the memory and processor are set.

[0047] In a ninth aspect, a chip is provided, including a processor for retrieving and executing program instructions or code stored in a memory, causing a communication device equipped with the chip to perform the methods described in the preceding aspects.

[0048] In a tenth aspect, another chip is provided, comprising: an input interface, an output interface, a processor, and a memory, wherein the input interface, the output interface, the processor, and the memory are connected via an internal connection path, and the processor is used to execute code in the memory, and when the code is executed, the processor is used to perform the methods in the above aspects.

[0049] It should be understood that the multimodal processing device mentioned in the third aspect and the model training device provided in the fourth aspect can be the chip mentioned in the ninth or tenth aspect, or it can also be the device mentioned in the fifth aspect. The beneficial effects achieved by the technical solutions and corresponding possible implementations of the second to tenth aspects of this application can be found in the above description of the technical effects of the first aspect and its corresponding possible implementations, and will not be repeated here. Attached Figure Description

[0050] Figure 1 A statistical chart of training data provided in an embodiment of this application;

[0051] Figure 2A schematic diagram of an implementation environment provided for an embodiment of this application;

[0052] Figure 3 This is a schematic diagram of another implementation environment provided for an embodiment of this application;

[0053] Figure 4 This is a schematic diagram illustrating another implementation environment provided in the embodiments of this application;

[0054] Figure 5 A flowchart illustrating a multimodal processing method provided in this application embodiment;

[0055] Figure 6 A mapping diagram between DP nodes and training data is provided in an embodiment of this application;

[0056] Figure 7 This is a schematic diagram illustrating how a DP node processes training data, as provided in an embodiment of this application.

[0057] Figure 8 A schematic diagram illustrating a data splicing method provided in an embodiment of this application;

[0058] Figure 9 This is a schematic diagram illustrating the movement of non-text data as provided in an embodiment of this application.

[0059] Figure 10 A schematic diagram illustrating a model training process provided in an embodiment of this application;

[0060] Figure 11 A schematic diagram of a model training architecture provided for an embodiment of this application;

[0061] Figure 12 This is another schematic diagram illustrating the movement of non-text data provided in the embodiments of this application;

[0062] Figure 13 This is a schematic diagram illustrating the adjustment of training data provided in an embodiment of this application;

[0063] Figure 14 This is a schematic diagram illustrating another adjustment of training data provided in an embodiment of this application;

[0064] Figure 15 This is a schematic diagram illustrating another type of training data adjustment provided in an embodiment of this application;

[0065] Figure 16 This is another schematic diagram illustrating the adjustment of training data provided in an embodiment of this application;

[0066] Figure 17 A structural flowchart of a multimodal processing device provided in an embodiment of this application;

[0067] Figure 18 This application provides a schematic diagram of the structure of a network device according to an embodiment of the present application.

[0068] Figure 19 This is a schematic diagram of another network device provided in an embodiment of this application. Detailed Implementation

[0069] The terminology used in the embodiments section of this application is only for explaining specific embodiments of this application and is not intended to limit this application. To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be further described in detail below with reference to the accompanying drawings.

[0070] With the development of machine learning technology, large models (LMs) with large parameters and complex computational structures have been widely used in various fields, including natural language processing, computer vision, speech recognition, and recommendation systems. In some cases, large models that support processing multiple types of data for application in multiple fields can also be called multimodal models. Before using them for multimodal processing of multiple types of data, the models are trained to obtain large models suitable for multimodal processing.

[0071] During the training of large models, model training efficiency and load balancing are related to the padding ratio. Since large models are used to process multiple types of data, the datasets used to train large models also include multiple types of training data, and these different types of training data have varying lengths. Figure 1 This is a statistical diagram of training data provided in an embodiment of this application. The text subsequence indicates training data of text type, size indicates length, #image subsequence indicates the number of image blocks included in the image subsequence, and density indicates proportion. See also... Figure 1 (1) The length of the training data in the dataset used to train the large model is highly discrete, that is, the length of multiple training data is not uniform, and the length of different training data is very different.

[0072] If training with training data of varying lengths is chosen under dynamic shape conditions, although the amount and proportion of padding are small, load balancing is difficult to achieve. For example, the training data in the dataset is divided into multiple micro-batches, which are trained in parallel, such as through pipeline parallelism (PP) or virtual pipeline parallelism (VPP). If the lengths of the training data in each micro-batch are different, the training time for different micro-batches will vary, resulting in poor load balancing, increased vacuolation during the training of large models, and low training efficiency. Adding padding to unify the length of different training data to train under a fixed static shape can improve load balancing and reduce vacuolation, but the increased padding bits lead to wasted computational resources and low training efficiency.

[0073] Furthermore, since training large models involves training on various types of training data simultaneously, the amount of non-textual data within these different types of training data will vary. Training data types include, for example, image-text pairs, video understanding, image-text interleaving, optical character recognition (OCR) recognition, object detection, plain text, and speech understanding. These datasets exhibit differences in the distribution of non-textual data. See also... Figure 1 (2) Taking non-text images as an example, the number of image patches in the training data of the dataset used to train large models is highly discrete. The number of image patches is determined based on the number of images and image resolution in the training data. If the number of image patches in different micro-batches is different, it will lead to uneven distribution of computing resources and different encoding tasks in different micro-batches. Some micro-batches will have a large number of image patches to be encoded, while some micro-batches will have a small number of image patches to be encoded, which will affect load balancing and reduce the efficiency of model training. Based on this, a multimodal processing method is urgently needed to perform multimodal processing using the trained model.

[0074] In related technologies, an offline assembly scheme is employed before training a model for multimodal processing. For the dataset used to train the model, offline assembly is manually performed before using the dataset to train a large model. First, the training data in the dataset is read, and then assembled according to the length of the read training data or the number of image patches it includes. During the assembly process, it is considered whether the number of image patches or tokens after assembly meets a load balancing condition. Here, the number of tokens refers to the sequence length of the assembled training data. Meeting the load balancing condition can be understood as ensuring that the number of image patches and the number of tokens are as similar as possible. By assembling the training data offline, the length of the training data or the number of image patches used in training large models is standardized. The assembled training data is then used for model training, thereby achieving load balancing during the training process.

[0075] However, the aforementioned offline assembly method requires reassembling the training data in any dataset that changes. For example, if a specific training data point is altered, the proportions of training data change, a new type or modality of dataset is introduced, or the tokenization rules are adjusted (tokenization rules include image segmentation rules, segmentation methods, number of segments, and the number of tokens representing each sub-image), the training data must be reassembled. This multiple offline assembly processes consume significant computational and storage resources. Furthermore, training data assembled based on balanced conditions may not be evenly distributed across different training data types during subsequent sampling, reducing the stability of large model training and potentially leading to poorer initial model performance.

[0076] This application provides a multimodal processing method. Please refer to the embodiments provided. Figure 2 The diagram illustrates an implementation environment for the method provided in this application embodiment. This environment includes a computing device 01, which can be used to execute the multimodal processing method provided in this application embodiment. The computing device 01 acquires a first model trained using third and fourth training data, and uses the first model to perform multimodal processing on the first data to obtain a multimodal processing result. The third and fourth training data are obtained by adjusting the length discrepancy between the first and second training data. Optionally, the computing device 01 can also execute a model training method to train the first model based on the third and fourth training data.

[0077] Optionally, the multimodal processing method and model training method provided in this application can be executed by one computing device 01, or by multiple computing devices 01 interactively. Figure 3 For example, Figure 3Both the first and second equipment belong to Figure 2 The computing device 01 in the application establishes a communication connection between the first device and the second device via a wired or wireless means. The first device is used to execute a model training method and send the trained first model to the second device. The second device is used to execute the multimodal processing method provided in the embodiments of this application and use the first model to perform multimodal processing on the first data.

[0078] In addition, multiple computing devices 01 can also have other interactions, continuing to... Figure 3 For example, the first device acquires first training data and second training data, sends the first and second training data to the second device, the second device loads the received first and second training data, and executes a model training method after loading the first and second training data to train a first model. In this case, the computing device that uses the first model to execute the multimodal processing method provided in the embodiments of this application can be... Figure 3 The second device can also be a device different from the second device, such as the first device or a third device that is different from both the first and second devices.

[0079] Figure 4 This is a schematic diagram of another implementation environment provided by an embodiment of this application. Figure 4 The first device 10, the second device 11, and the third device 12 all belong to Figure 2 The computing device 01 is described above. A first device 10 acquires first training data and second training data, sends the first and second training data to a second device 11, which loads the first and second training data, executes a model training method to train a first model, and deploys the first model on the second device 11. Subsequently, a third device 12 accesses and uses the first model deployed on the second device 11, executing the multimodal processing method provided in this embodiment.

[0080] Optionally, computing device 01 can be any device with data processing capabilities. Computing device 01 can be a server, such as a central server, an edge server, or a local server in a local data center. The server can be a physical server or a cloud server providing cloud computing services. In some embodiments, computing device 01 can also be a terminal device such as a desktop computer, laptop computer, or smartphone. Besides being a standalone device, computing device 01 can also be a component within a device, such as a transceiver, processor, or chip.

[0081] This application provides a multimodal processing method, which can be applied to... Figures 2-4 The implementation environment shown, taking the method executed by a computing device as an example, is illustrated in the flowchart below. Figure 5 As shown, it includes S501-S502.

[0082] S501, obtain the first data to be processed.

[0083] For example, the first data can be any data to be processed in a multimodal manner. Multimodal processing includes multimodal understanding and multimodal generation. Multimodal generation refers to mapping and fusing data from multiple different modalities to generate new data. Multimodal understanding refers to extracting and fusing information from data from multiple different modalities to gain a deeper understanding and inference of the meaning of the data. Different modalities include, for example, vision, hearing, and natural language. Multimodal understanding includes, but is not limited to, natural language processing, computer vision, and speech recognition. Natural language processing includes speech recognition, semantic analysis, corpus construction, text classification, machine translation, sentiment analysis, and text-to-image processing. Computer vision includes image classification, object detection, semantic segmentation, optical character recognition, visual question answering, image description, and visual localization. Speech recognition is used to convert speech signals into text or commands.

[0084] Multimodal processing may encompass various processing tasks, and the primary data can also be of various types, such as image-text pairs, video understanding, image-text interleaving, OCR recognition, object detection, plain text, and speech understanding. Image-text pairs refer to paired image and text data, where the images and text provide complementary information. Video understanding includes video classification, temporal action detection, spatial action detection, and the combination of video and speech; training data for video understanding can be videos or consecutive image frames from videos. Image-text interleaving refers to data composed of multiple interleavings of images and text, such as images embedded within text context. OCR recognition is used to identify and parse text information in images; primary data for OCR recognition includes images and their corresponding text, indicating words and distractions within the image. Object detection is used to identify and locate objects in images; primary data for object detection includes video data or image data and corresponding annotation information. Speech understanding refers to the automatic recognition and semantic understanding of sentences; primary data for speech understanding includes speech data.

[0085] This application does not limit the method of obtaining the first data. The first data can be data collected by a computing device. Taking multimodal processing including visual question answering as an example, the first data includes image data. If the computing device includes an image acquisition device, the computing device can acquire images as the first data to be used for visual question answering. In one possible case, the first data can also be data obtained by the computing device accessing storage space. The accessed storage space can be the storage space of the computing device itself or the storage space of a device that establishes a communication connection with the computing device. Continuing with the example of image data as the first data, after the computing device acquires an image using the image acquisition device, it stores the acquired image in local storage space. Then, based on the multimodal processing requirements, it selects an image from the images stored in the storage space as the first data. Optionally, the computing device can also access the Internet to obtain the first data. For example, it can search for images through a browser and use the searched images as the first data.

[0086] S502, the first model performs multimodal processing on the first data to obtain the multimodal processing result. The first model is trained using the third and fourth training data.

[0087] Regardless of how or what type of first data the computing device acquires, a first model can be used to perform multimodal processing on the first data. The first model can be any multimodal processing model used for human-computer interaction, and it can be any deep learning framework (e.g., PyTorch, MindSpore, TensorFlow, Jax). PyTorch is a machine learning model used for applications such as image recognition and language processing; MindSpore is a full-scene artificial intelligence (AI) framework; TensorFlow is an open-source machine learning framework used to build and train various machine learning models; and Jax is a machine learning and numerical computation framework used for high-performance computing. Optionally, the first model can also be accelerated using any distributed acceleration framework. Taking PyTorch as an example, the distributed acceleration framework used could be Megatron or Deepspeed.

[0088] For example, the first model can be a model trained by a computing device, or it can be a model trained by a different training device. In this case, the computing device can receive the model parameters of the first model sent by the training device to obtain the first model, or it can access the storage space of the training model to obtain the first model. Since the processes of obtaining the first model by the computing device and the training device are similar, the following will take the model training method performed by the computing device as an example to illustrate the process of obtaining the first model by the computing device through model training. This model training method includes steps one through three.

[0089] Step 1: Obtain the recorded first and second training data.

[0090] For example, the computing device acquires multiple training data sets required for training the first model based on model training needs. Since the first model can be any machine learning model used to perform multimodal processing, the training data for training the first model can also be of various types. The types of training data are similar to the types of the first data, and can be found in the relevant descriptions of the above embodiments, which will not be repeated here.

[0091] This application does not limit the process of acquiring training data. The computing device can provide an information input control to receive training data input by a user, where the user refers to the object using the computing device to train the model. The computing device can also access open-source datasets to obtain training data, and, if access permissions are granted, it can also access non-open-source datasets to obtain training data. Non-open-source datasets may include, for example, internal enterprise databases or other datasets subject to access control requirements. The computing device can also access the Internet to search for and obtain training data via the network.

[0092] Regardless of the method or type of training data acquired by the computing device, the training data can be initialized by sampling and loading it to obtain the loaded training data. Since the access speed of computer memory is higher than that of hard drives and other storage devices in the computing device, loading the training data into computer memory can improve the performance and response speed of the computing device. In one possible scenario, in addition to data loading, the initialization phase of the training data can also include data aggregation after loading. Aggregation combines training data from different sources or with different characteristics to improve the model's generalization ability and training effect.

[0093] After completing the initialization phase of the training data, the computing device can perform the operations of the training phase, specifically steps two and three, to adjust the first reference difference between different training data sets to improve model training efficiency. The first reference difference indicates the length dispersion of different training data sets, such as the degree of dispersion of the lengths of different training data sets relative to their average length. Optionally, the first reference difference can be the relative standard deviation of the lengths of different training data sets, or other values ​​that reflect the degree of dispersion relative to the average value. The magnitude of the first reference difference is positively correlated with the degree of dispersion; the smaller the first reference difference, the lower the degree of dispersion of the lengths of different training data sets relative to their average length, meaning the lengths of the training data are closer to the average length.

[0094] Optionally, the computing device can adjust the first reference difference between a portion of the training data in multiple sets of training data to reduce the shape differences between the multiple sets of training data. Alternatively, the computing device can adjust the first reference difference between all the training data in multiple sets of training data to achieve shape uniformity across the multiple sets of training data.

[0095] In the case where the computing device uses DP nodes to process training data during the training process of obtaining the first model, a set of training data can be understood as the training data processed by DP nodes in a batch. Figure 6 This application provides a correspondence between DP nodes and training data in its embodiments. Figure 6 The image shows four training data sets, designated as training data #1, #2, #3, and #4. The four training data sets are designated as DP1, DP2, DP3, and DP4. Since... Figure 6 If a DP node processes one training data point, then a set of training data refers to one training data point. If a DP node processes multiple training data points, then a set of training data refers to multiple training data points.

[0096] Since the process of adjusting the first reference difference between different training data is similar, we will use the first and second training data as examples to illustrate the process of steps two through three. Furthermore, the prefixes "first" and "second" before "first training data" and "second training data" are intended to distinguish different training data, not to limit the training data being adjusted. The first and second training data can correspond to the same DP node, meaning they are data processed at different times on the same DP node. Alternatively, the first and second training data can correspond to different DP nodes, meaning they are data processed at the same time using different DP nodes.

[0097] against Figure 6In this context, one training data point corresponds to one DP node. A first training data point can be understood as one training data point, and a second training data point can be understood as another training data point. For example, the first training data point is... Figure 6 The training data corresponding to DP1 is #1, and the second training data is... Figure 6 The training data corresponding to DP4 in the example is #4. Furthermore, the first and second training data adjusted in this embodiment can be one or more, such as the scenario of adjusting all training data as described in the above embodiment.

[0098] Step 2: Adjust the first reference difference between the first training data and the second training data to obtain the third training data and the fourth training data. The first reference difference indicates the length dispersion of the first training data and the second training data. The second reference difference between the third training data and the fourth training data is less than the first reference difference between the first training data and the second training data.

[0099] In some cases, if the lengths of the first and second training data differ, it may affect model training efficiency. The length of the first training data refers to the total length of all data types included in the first training data. For example, if the first training data is plain text, its length is the length of the text sequence; if it's image-text pairs, its length is the sum of the lengths of the image and text data. Furthermore, if the first training data is a single piece of data, its length is the number of bytes occupied by that piece. If the first training data is a dataset containing multiple training data pieces, its length is the sum of the number of bytes occupied by each training data piece.

[0100] Because computing devices synchronize the training time of each DP node during the parallel training of the model using multiple DP nodes, Figure 7 This is a schematic diagram of training data provided in an embodiment of this application. Figure 7 The # symbol indicates training data. Figure 7 Four training data sets are shown: training data #1, training data #2, training data #3, and training data #4. Training data #4 is the longest. Therefore, even if the other three DP nodes have processed training data #1, #2, and #3, completing one round of training, they will wait for the fourth DP node to finish processing training data #4 before starting the next iteration. The waiting period for the three DP nodes can be understood as computing power being used to process patches. Based on... Figure 7It is known that if the lengths of the first and second training data are different, it will waste the computing power of the computing nodes, prolong the training time of the computing nodes, and thus lead to low model training efficiency.

[0101] The difference between the first length corresponding to the first training data and the first length corresponding to the second training data may also affect the model training efficiency. Here, the first length refers to the length of non-text data, which has encoding requirements and includes elements such as images, audio, or video. The first length corresponding to the first training data refers to the length of the non-text data within the first training data. For example, if the first training data has not undergone concatenation, the training data processed by one DP node in one iteration constitutes the first training data. Furthermore, the first length corresponding to the first training data can also indicate the length of the non-text data in the first and fifth training data. For example, if the first training data has undergone concatenation, the training data processed by one DP node in one iteration constitutes the concatenated first training data, and the fifth training data is used to concatenate with the first training data.

[0102] Furthermore, if the first training data in step one can be a single training data set, then the non-text data in the first training data can be a sub-data set belonging to the non-text type within the first training data set, representing a portion of the data in the first training data set. Alternatively, if the first training data in step one can be a dataset containing multiple training data sets, then the non-text data in the first training data can be any training data set belonging to the non-text type within the first training data set.

[0103] For example, the length of non-text data indicates the amount or quantity of non-text data; if the non-text data is image data, the length of non-text data indicates the number of image blocks; if the non-text data is video data, the length of non-text data indicates the number of video frames; and if the non-text data is audio data, the length of non-text data indicates the number of audio segments.

[0104] Optionally, the first length corresponding to the second training data has a similar meaning to the first length corresponding to the first training data. For example, the first length corresponding to the second training data indicates the length of non-text data in the second training data, or the first length corresponding to the second training data indicates the length of non-text data in both the second and sixth training data. For a description of the first length corresponding to the second training data, please refer to the description of the first length corresponding to the first training data in the above embodiments, which will not be repeated here.

[0105] In one possible scenario, if the lengths of non-textual data types in the training data processed by DP nodes differ, it can lead to low model training efficiency. Taking model training based on two DP nodes as an example, DP node 1 corresponds to the first training data, and DP node 2 corresponds to the second training data. DP node 1 has 4 non-textual data types, while DP node 2 has 10. DP node 1 needs to encode 4 non-textual data types, while DP node 2 needs to encode 10. The different numbers of non-textual data types encoded by different DP nodes result in uneven resource allocation, affecting the encoding efficiency of non-textual data and leading to low model training efficiency.

[0106] Based on this, the computing device can adjust the first reference difference between the length of the first training data and the length of the second training data to improve model training efficiency. The computing device can also adjust the first reference difference between the first length corresponding to the first training data and the first length corresponding to the second training data to improve model training efficiency. Furthermore, the computing device can adjust the first reference difference corresponding to the data length and the first reference difference corresponding to the first length to improve model training efficiency. The following examples illustrate the process of adjusting different first reference differences.

[0107] Adjustment process one: The first reference difference is the reference difference between the data length of the first training data and the data length of the second training data.

[0108] In the first adjustment process, the third training data is obtained based on the spliced ​​first training data, and the fourth training data is obtained based on the spliced ​​second training data. The spliced ​​first training data is obtained based on the fifth training data and the first training data, and the spliced ​​second training data is obtained based on the sixth training data and the second training data.

[0109] For example, the computing device acquires fifth training data and sixth training data; it concatenates the fifth training data and the first training data, and concatenates the sixth training data and the second training data, wherein the data length of the concatenated first training data is equal to the sum of the data lengths of the fifth training data and the first training data, and after the data length of the concatenated first training data is equal to the data lengths of the sixth training data and the second training data, the second reference difference between the data length of the concatenated first training data and the data length of the concatenated second training data is less than the first reference difference.

[0110] In this context, the fifth training data point refers to training data that differs from both the first and second training data points, and the sixth training data point also refers to training data that differs from both the first and second training data points. Furthermore, the fifth and sixth training data points can be a single data point or multiple data points, and vice versa. Additionally, both the fifth and sixth training data points can include complete training data or partial training data. Since the acquisition and concatenation processes for the fifth and sixth training data points are similar, the following explanation will use the fifth training data point as an example to illustrate the process of acquiring the fifth training data point and concatenating it with the first training data point.

[0111] In one possible scenario, the computing device may refer to the type of the first training data when selecting the fifth training data. For example, the computing device acquires multiple data piles, each containing at least one loaded training data, and the multiple data piles are divided based on the type of training data; a data pile is selected from the multiple data piles based on the type of the first training data, and the fifth training data is acquired from the selected data pile.

[0112] Optionally, the computing device can partition the loaded training data, grouping training data of the same type into multiple data piles. For example, after loading multiple training data, the computing device may first partition the training data into piles, and then merge the training data from the multiple data piles. Alternatively, the computing device may directly acquire multiple data piles; for example, the multiple training data acquired by the computing device may be training data that has already been partitioned into piles based on type.

[0113] Regardless of how the computing device acquires multiple data piles, it can determine the type that does not conflict with the first training data based on the type of the first training data, and then select a data pile from the multiple data piles based on the determined type. Continuing with the example of training data types including type A, type B, type C, type D, and type E, the first training data is type A. Type A and type B can be concatenated together, and types C, D, and E can be concatenated together. That is, the type that does not conflict with type A is type B. The computing device selects a data pile from the multiple data piles that includes training data of type B and type A, and then selects the fifth training data from the selected data pile.

[0114] Alternatively, during the data concatenation process, the computing device employs a strategy of concatenating training data of the same type together. That is, the first training data can only be concatenated with training data of the same type. In this case, the computing device selects a data pile containing training data of type A from multiple data piles, and then selects the fifth training data from that pile. By concatenating the first training data with the fifth training data of the specified type, the accuracy and performance of the model training are improved.

[0115] Optionally, the computing device may also disregard the type of the first training data and randomly select a fifth training data of any type different from the first and second training data from the loaded training data. That is, the computing device employs a full joint concatenation strategy, where training data of one type can be concatenated with training data of any type. For example, if the loaded training data includes types A, B, C, D, and E, regardless of the type of the first training data, it can be concatenated with any one or more types of training data. Therefore, the computing device can randomly select the fifth training data.

[0116] For example, the computing device can selectively select the fifth training data based on the type of the first training data, or selectively select the fifth training data based on the data length of the first training data. For instance, the computing device obtains a reference length, which indicates the expected length after concatenating the first training data. The reference length can be the longest length among the training data to be adjusted. Taking 100 training data items loaded as an example, where training data 1 is the first training data, training data 2 is the second training data, and training data 10 has the longest data length among the 100 training data items, then the reference length can be the data length of training data 10. The reference length can also be the length of the training data that the DP node supports processing; this embodiment of the application does not limit this.

[0117] After obtaining the reference length, the computing device calculates the difference between the reference length and the length of the first training data, and selects one or more training data points as the fifth training data point based on the difference. Alternatively, the computing device can select the fifth training data point based solely on the reference length, solely on the type of the first training data, or both the reference length and the type of the first training data. For example, it can select the fifth training data point from the selected data set based on the reference length. Selecting the fifth training data point based on the reference length ensures that the concatenated first training data fills the entire sequence length, thereby reducing the amount of padding in the concatenated first training data.

[0118] Regardless of how the computing device selects the fifth training data, the fifth training data and the first training data can be concatenated together. The fifth training data can be concatenated before or after the first training data. Furthermore, if the fifth training data includes multiple training data points, a portion of the fifth training data can be concatenated before the first training data, and another portion after.

[0119] Figure 8This is a schematic diagram of training data splicing provided in an embodiment of this application. Figure 8 Training data #1 corresponds to the first training data, and training data #4 corresponds to the second training data. See [link / reference] Figure 8 The fifth training data, concatenated with the first training data, includes multiple training data sets: complete data from training data #2 and partial data from training data #3. This fifth training data set is appended after the first training data set. The sixth training data, concatenated with the second training data set, consists of a single training data set, which is another portion of training data #3.

[0120] In addition to reducing the first reference difference corresponding to the data length by splicing training data, computing devices can also splice patches. Continuing with... Figure 8 For example, since there is still a gap between the second training data and the reference length after concatenation with training data #3, the computing device further concatenates the second training data and the patch together based on training data #3. Regardless of the method used by the computing device to concatenate the training data, the second reference difference between the data length of the concatenated first training data and the data length of the concatenated second training data is smaller than the first reference difference between the data length of the first training data and the data length of the second training data.

[0121] Continue with Figure 8 For example, Figure 8 The first training data is training data #1, and the second training data is training data #4. Figure 8 The first training data after splicing is the first row of data, which includes training data #1, training data #2 and training data #3. Figure 8 The second training data after splicing is the second row of data, which includes training data #3, training data #4 and a patch. The spliced ​​first training data and the spliced ​​second training data have the same data length. The relative standard deviation calculated based on the data length of the first row of data and the data length of the second row of data is less than the relative standard deviation calculated based on the data length of training data #1 and the data length of training data #4.

[0122] In one possible scenario, after concatenating the first and fifth training data sets, the computing device may also generate a cross-sample attention mask to ensure that different training data sets in the concatenated first training data set do not interfere with each other. This process of concatenating training data and then generating the attention mask can also be called packing.

[0123] Adjustment process two: The first reference difference is the reference difference between the first length corresponding to the first training data and the first length corresponding to the second training data.

[0124] The first length corresponding to the first training data is determined based on the non-text type data in the first training data, indicating the length of the non-text type data corresponding to the first training data. The non-text type data corresponding to the first training data refers to the non-text type data on the DP node corresponding to the first training data. If the computing device does not perform adjustment process one and concatenates the first training data and the fifth training data together, then the non-text type data corresponding to the first training data is also the non-text type data in the first training data. If the computing device performs adjustment process one and concatenates the first training data and the fifth training data together, then the non-text type data corresponding to the first training data refers to the non-text type data included in the data concatenation result, that is, the non-text type data in the first training data and the non-text type data in the fifth training data. Regardless of the situation of the non-text type data corresponding to the first training data, the following explanation will assume that the number of non-text type data corresponding to the first training data is less than the number of non-text type data corresponding to the second training data, that is, the first length corresponding to the first training data is less than the first length corresponding to the second training data, to illustrate the process of adjusting the first reference difference corresponding to the first length.

[0125] Figure 9 This is a schematic diagram illustrating a process for adjusting a first length, as provided in an embodiment of this application. See also... Figure 9 The computing device did not perform the adjustment process on the first training data and the second training data. The training data processed by the DP0 node is the first training data belonging to the image-text pair, and the training data processed by the DP1 node is the second training data belonging to the video understanding. The first training data is one training data and the second training data is one training data. Figure 9 A rectangle in the image represents a sub-data element, and letters are used to indicate sub-data that belongs to text and does not require encoding. In the case of non-text types such as image types, ... Figure 9 The different filled rectangles in the image are designed to distinguish different images. For example, in video understanding, shaded rectangles and white rectangles are used to segment images in different video frames. Figure 9 The number of non-text type data corresponding to the first training data is 1, the number of non-text type data corresponding to the second training data is 6, and the first length corresponding to the first training data is less than the first length corresponding to the second training data.

[0126] In cases where the first length corresponding to the first training data is less than the first length corresponding to the second training data, the third training data and the first training data correspond to the same DP node, and the fourth training data and the second training data correspond to the same DP node, the computing device can select non-text type data from the non-text type data on the DP node corresponding to the second training data; move the selected non-text type data to the DP node corresponding to the first training data, determine the data on the DP node corresponding to the second training data after the move as the fourth training data, determine the data on the DP node corresponding to the first training data after the move as the third training data, and the second reference difference between the first length corresponding to the fourth training data and the first length corresponding to the third training data is less than the first reference difference.

[0127] The computing device can select non-text type data from the DP node corresponding to the second training data based on the first length corresponding to the first training data and the first length corresponding to the second training data. Continuing... Figure 9 For example, the computing device determines that there are a total of 7 non-text type data corresponding to the first training data and the second training data based on the first length corresponding to the first training data and the first length corresponding to the second training data. Dividing 7 by 2, we get 3.5. Thus, we determine that the number of non-text type data in the third training data obtained by moving is 3 to 4. The computing device selects 3 non-text type data from the non-text type data corresponding to the second training data and moves them to the DP node corresponding to the first training data.

[0128] In one possible scenario, when selecting non-text data, the computing device may also move related non-text data as a whole. This related non-text data could belong to the same original image, video frame, or audio sample. Figure 9 Sub-data that use the same padding, for example Figure 9 The dashed boxes in the diagram above identify three non-text data types. Because the encoding processes of related non-text data types can affect each other, these related non-text data types must be encoded using the same DP node. Figure 9 The three non-text data types marked by the dashed boxes in the diagram should correspond to the same DP node. That is, if the computing device moves three non-text data types, all three must be moved together from DP1 to DP0. Figure 9 The following figure shows the result of the shift. By shifting non-text data, the amount of non-text data in the third and fourth training data is balanced, thereby achieving dynamic load balancing of non-text data processed by different DP nodes without the need for preprocessing and bucketing.

[0129] Optionally, the computing device may choose to perform only one adjustment process—reducing the first reference difference between the data lengths of the first and second training data—to obtain third and fourth training data. For example, concatenating the first and fifth training data yields concatenated first training data; generating a cross-sample attention mask on the concatenated first training data yields the third training data. Concatenating the first and sixth training data yields concatenated second training data; generating a cross-sample attention mask on the concatenated second training data yields the fourth training data.

[0130] The computing device may also choose to perform only adjustment process two, reducing the first reference difference between the first length corresponding to the first training data and the first length corresponding to the second training data, to obtain the third and fourth training data. For example, after moving the non-text data in the second training data to the non-text data in the first training data, the moved first training data is used as the third training data, and the moved second training data is used as the fourth training data.

[0131] Alternatively, the computing device can choose to comprehensively execute adjustment process one and adjustment process two, further adjusting the first reference difference corresponding to the first length based on the first reference difference corresponding to the adjusted data length. For example, the computing device concatenates the first training data and the fifth training data to obtain the concatenated first training data, generates a cross-sample attention mask on the concatenated first training data, and obtains the assembled first training data. Concatenates the first training data and the sixth training data to obtain the concatenated second training data, generates a cross-sample attention mask on the concatenated second training data, and obtains the assembled second training data. Non-textual data in the assembled second training data is moved after the non-textual data in the assembled first training data, and the moved first training data is used as the third training data, and the moved second training data is used as the fourth training data.

[0132] Next, we will combine... Figure 10 Illustrate the process of the computing device performing the first and second adjustment processes. Figure 10 The black rectangles in the diagram indicate operations performed during the training phase, and the white rectangles indicate operations performed during the initialization phase. See also... Figure 10As shown in the diagram below, during the initialization phase of this application, the computing device loads multiple training data sets used for model training and groups these training data sets into multiple data piles based on their types. These data piles are then merged. Next, the computing device determines the assembly strategy for the merged training data, identifies the first, second, fifth, and sixth training data sets to be assembled, loads pointers to these data sets, and retrieves the first and second training data sets from the loaded training data based on these pointers. According to the assembly strategy, the device performs the assembly process in adjustment step one on the retrieved first and second training data sets, assembling the first and fifth training data sets to obtain the first assembly result, and then assembling the second and sixth training data sets to obtain the second assembly result. Since the assembly result represents training data processed by a DP node within a batch, in some cases, the assembly result can be referred to as a training sample or a set of training data.

[0133] The computing device performs iterative training based on the first and second assembly results, and executes an adjustment process two during the iterative training process. This process adjusts the non-textual data in the first and second assembly results to obtain the third and fourth training data. This adjustment process corresponds to... Figure 10 Training of a multi-mode equilibrium model. Based on... Figure 10 As can be seen, the above model training method restricts data bucketing in the initial stage, meaning that the first and second training data are training data without data bucketing. Data bucketing is used to sample training data that meets the equilibrium condition. Since this application balances the data length or at least one of the first and second training data by adjusting the first reference difference after loading and merging the first and second training data, equilibrium can be achieved without using data bucketing for sampling based on the equilibrium condition. By restricting the execution of data bucketing, uniform, random, non-repetitive, and wasteful sampling is ensured, resulting in high data utilization.

[0134] Step 3: Train the first model based on the third and fourth training data. The first model is used for multimodal processing.

[0135] In one possible implementation, the computing device performs iterative training based on the loaded training data. If the computing device concatenates multiple loaded training data before model training, for example, concatenating the first and fifth training data into one set of training data, and concatenating the second and sixth training data into one set of training data, the total number of training data used for model training will change, and the actual number of tokens included in each set of training data will also change. For example, the number of tokens in the first concatenation result obtained by concatenating the first and fifth training data may be greater than the number of tokens in the first training data.

[0136] If the total number of iterations is still determined based on the training data before splicing, overfitting and discrepancies between the learning rate curve and the actual situation may occur. Therefore, the computing device can obtain the total number of iterations corresponding to the third and fourth training data by adjusting the data length of the first and second training data to obtain the first reference difference, that is, the first reference difference includes the reference difference between the data length of the first and second training data. Based on the total number of iterations, the first model is obtained by iterative training using the third and fourth training data. The total number of iterations indicates the number of times the first model is obtained through iterative training.

[0137] Optionally, the third and fourth training data obtained by adjusting the first reference difference between the data length of the first and second training data can mean that the third and fourth training data are training data obtained by performing only adjustment process one, or that the third and fourth training data are training data obtained by performing both adjustment process one and adjustment process two.

[0138] Regardless of whether the process of obtaining the third and fourth training data involves adjusting the first reference difference corresponding to the first length, the computing device can further obtain the total number of iterations after performing the adjustment process one. Optionally, the computing device can use the iterative estimation module to select a portion of the concatenated training data for estimation to obtain the total number of iterations. The computing device can also provide an information input control to receive the total number of iterations input by the user. For example, since the size, type, and variance of the dataset used in actual training tasks may be very large, the iterative estimation module can only sample a portion of the training data for estimation to avoid affecting training time. However, sampling a portion of the training data will cause a difference between the estimated number of steps and the number of steps expected by the user. Therefore, the total number of iterations can also be determined by the user based on the concatenation of the training data or by using offline statistical methods. By re-obtaining the total number of iterations, the total number of iterations is reduced to the actual number of steps, ensuring that the learning rate curve conforms to the actual situation.

[0139] After determining the total number of iterations, the computing device can begin iterative training. Based on the total number of iterations, it iteratively trains using the third and fourth training data to obtain the first model. For example, multiple sets of training data are used for multiple rounds of iterative training until the total number of iterations is reached, resulting in the first model. The multiple sets of training data include the third and fourth training data. These multiple sets of training data can consist entirely of adjusted third and fourth training data; for example, the computing device performs step two on any two loaded training data sets. Alternatively, the multiple sets of training data can consist partially of adjusted third and fourth training data; for example, the computing device performs step two on a portion of the loaded training data.

[0140] Furthermore, the first model trained based on multiple sets of training data can be obtained by fine-tuning the parameters of the second model, or by training a new first model. The model training methods employed include, but are not limited to, pre-training, supervised fine-tuning (SFT), alignment training, and other model training methods. Alignment training, for example, includes reinforcement learning from human feedback (RLHF) and direct preference optimization (DPO).

[0141] Next, for the time being... Figures 11-14 For example, this illustrates the process of adjusting the first reference difference during the training phase of the computing device, and the model training process based on the model data adjusted to the first reference difference. See also... Figure 11 The loaded dataset (DS) includes datasets A, B, C, D, and E. Each dataset is split into four data splits. The computing device merges datasets A and B, resulting in a dataset whose length is the sum of the lengths of datasets A (len) and B, equal to 3.2 mega (M). The dataset length indicates the amount of training data included in the dataset.

[0142] Next, the computing device caches the merged dataset into a 0-buffered dataset (bufferDS) for data buffering. The 0-bufferDS buffer pool is 3.2MB in size, and the cached dataset is also 3.2MB in length. During the dataset buffering process, the computing device executes... Figure 10The data assembly operation shown determines the assembly strategy of training data in multiple data heaps under each DP node, and calculates the self-assembled dataset that can be obtained based on the assembly strategy. The training data included in the self-assembled dataset has a uniform data length, which is the reference length in the above embodiment, and in some cases can be called the given sequence length.

[0143] The computing device performs a data pre-loop on the training data within the self-assembled dataset. This pre-loop enables iterative estimation, determining the total number of iterations required for training based on the training data in the self-assembled dataset. After the pre-loop, the computing device caches a portion of the training data in the dataset into a buffer pool, loading 960 kilo (k) training data points at a time. Then, the computing device performs multi-process initialization, equally distributing the cached training data across processes. The assembly processes of each DP node and data processing process are independent, ensuring uniform data sampling without introducing additional communication, thus guaranteeing the equivalence of the training process and preventing model degradation. After each process obtains training data from its corresponding subsample library, it selects training data from the dataset for assembly based on an assembly strategy. This involves concatenating the training data and generating a cross-sample attention mask to obtain the data assembly result. The attention mask ensures that different training data points in the data assembly result do not interfere with each other. Detailed information on assembling training data can be found in the adjustment process described in step two, and will not be repeated here. Optionally, the computing device also records an assembly pointer during the training data assembly process. The assembly pointer is, for example,... Figure 11 In the dp_rank_0:(0:1,1:0), the rank is used to indicate the training data that has been assembled at the current time.

[0144] After assembly is complete, the computing device enters the training phase and begins iterative training. The computing device uses a batch collator to retrieve the data assembly results of the current batch from the data assembly results. Each DP phase summarizes the training pointers in the current batch and stores them in a distributed recorder. This process is... Figure 10 The pointer record is shown.

[0145] Optionally, the iterative training performed by the computing device on the data assembly results includes forward encoding, backward decoding, and parameter optimization. During the iterative training of the current batch of data assembly results, the computing device also performs inter-DP group communication, that is, moving non-text data of the data assembly results processed by different DP nodes. See also Figure 12The data assembly results loaded on each DP node contain multiple image blocks in the data assembly results on DP0, while the data assembly results on other DP nodes do not contain image blocks. The computing device broadcasts the image blocks on DP0 to other DP nodes, and each DP node can be used to process one image block. Each DP node encodes the image block to obtain the encoded result of each image data. Optionally, each DP node also performs a step-record operation during the image block encoding process. Step-record means that a record is made after each training step. Afterwards, DP0 collects the encoded results processed by other DP nodes and saves them.

[0146] Figure 13 This is a schematic diagram illustrating another model training process provided in an embodiment of this application. Figure 13 The input samples are the data assembly results from the above embodiments. The data distribution in the input samples can be found in [reference needed]. Figure 14 The right figure in (1) is compared to Figure 14 In the left figure of (1), the training data length is uniform on different DP nodes.

[0147] Because the input samples contain image blocks, the computing device needs to encode these image blocks. Furthermore, since the number of image blocks loaded on different DP nodes varies, the computing device will... Figure 13 As shown, a greedy data balancing algorithm is used for multi-mode data distribution and communication, moving image data across different DP nodes to achieve a balance in the amount of image data across different DP nodes. Figure 14 The equilibrium result in the right figure of (2) and the moving process can be found in the relevant description of the adjustment process 2, which will not be repeated here.

[0148] also, Figure 9 and Figure 14 This example illustrates the movement of non-text data between DP nodes, demonstrating that computing devices employ algorithms for optimal communication planning before encoding, rather than limiting the methods for moving non-text data. Non-text data can be like... Figure 9 and Figure 14As shown, point-to-point communication can be used to move data from one DP node to another. Many-to-one or one-to-many communication can also be used, moving data from multiple DP nodes to one DP node, or from one DP node to multiple DP nodes. Taking a computing device using four DP nodes to process four sets of training data as an example, with each DP node processing 1, 4, 9, and 2 image blocks respectively, the computing device uses a one-to-many communication strategy to move the image blocks from the third DP node to the first, second, and fourth DP nodes respectively. After the move, each of the four DP nodes processes 4 image blocks.

[0149] The computing device performs forward encoding on the equalized image blocks using a multi-mode encoder. By moving the image blocks, it balances the number of image blocks processed by different DP nodes, compared to... Figure 14 The left image of (2) effectively shortens the time for DP node to encode image blocks, resulting in high computing power utilization. Optionally, after obtaining the encoding result, the computing device performs multi-mode data restoration communication to restore the moved image data back to the DP node before the movement, thus obtaining... Figure 14 The restoration result of (3). The computing device will Figure 14 The restoration result shown in (3) is used as the input feature to input the large language model for forward and backward propagation. The computing device also needs to decode the image block encoding results. After obtaining the backward gradient through backpropagation, the computing device will again perform multi-mode data distribution communication to unify the decoding workload of different DP nodes. The balanced backward gradient is then back-decoded by the multi-mode encoder, and the optimizer parameters are updated based on the decoding results to achieve one round of iterative training.

[0150] In one possible scenario, if model training is performed on a large-scale cluster, training interruptions may occur due to physical or human factors. Therefore, computing devices also record the model training status during the training process to support resuming training at breakpoints. For example, the first model is obtained by continuing training after the training interruption based on training pointers or identification information. The training pointers or identification information are saved based on the number of iterations during the model training process and are used to identify unused training data.

[0151] For example, taking the training pointer as an example, the computing device can save the training pointer at the current moment when the number of iterations in the current training step reaches the save count. The training pointer is used to continue training after the model training is interrupted. The save count can be any number set based on experience and the implementation scenario, such as 100, or other positive integers.

[0152] This application does not limit the stored training pointer. If the third and fourth training data used during the training of the computing device model were not obtained through adjustment process one, then one training data point is used per iteration. Therefore, the number of iteration training steps is associated with the training data. In the 4000th iteration training, the training data used is also the 4000th training data point. The training pointer, by indicating the current iteration training step number, can record the training data and thus identify unused training data.

[0153] If the third and fourth training data used by the computing device are obtained through adjustment process one, then one or more training data sets may be used at a time during iterative training. If only the number of iterations is recorded, it is impossible to determine the training data used. Therefore, the training pointer must indicate the unused training data at the current moment in order to record the training data and identify unused training data. Since the process of saving and using different training pointers is similar, the following example will illustrate the use of the training pointer indicating unused training data.

[0154] Optionally, the computing device can perform periodic saving at intervals. For example, after starting model training using multiple sets of training data, when the number of iterations at the current moment reaches the required number of saves, a checkpoint (ckpt) is saved. Continuing with the example of 100 saves, the computing device performs a checkpoint save when performing the 100th iteration using multiple sets of training data, saving the training pointer at the current moment. Then, model training continues, and when another 100 iterations are performed, a checkpoint save is performed again.

[0155] If the computing device performs only one iteration at a time, only one training pointer is saved; if the computing device performs multiple iterations in parallel at the same time, multiple training pointers are saved. See also Figure 15 , Figure 15 The training data consists of 16,000 sets from a single sample database. Because the computing device uses two data processes and two dynamic programming (DP) nodes to execute the training task in parallel, the sample database is divided into four sub-sample databases. Each sub-sample database corresponds to a different data process and DP node. For example... Figure 15 The first column in (1) corresponds to the DP node with DP rank = 0, and the second column corresponds to the DP node with DP rank = 1. Each subsample library is assigned four thousand sets of training data.

[0156] The computing device performs a checkpoint saving during the 100th iteration. In this case, the training pointer for the subsample library with DP level = 0 is (300, 360), and the training pointer for the subsample library with DP level = 1 is (240, 320). Here, 300 indicates that the current process has trained on the 300th training data set in the subsample library, and 360 indicates that the current process has trained on the 360th training data set in the subsample library.

[0157] For example, in addition to storing the training pointer, the computing device also stores the model parameters at the current moment, such as... Figure 10 The model saving and pointer saving shown are used to determine the training data to be used based on the training pointer in the event of a training interruption, and to adjust the saved model parameters based on the training data. By periodically recording the training pointer, the misuse of already trained training data is effectively avoided when retraining after an interruption, ensuring the accuracy of model training and thus achieving non-reuse and non-waste of training data. Taking the example of each DP node of the computing device retrieving training data from the subsample library for model training, each DP node summarizes the latest subsample library pointer in the batch, that is, the training pointer, into the distributed recorder (recoder). At the interval of reaching the save point (ckpt_interval), that is, the number of saves, the master node of the computing device collects the encoder data from each DP node and summarizes and saves it together with the model ckpt. After the breakpoint resume training task is launched, if encoder data is detected, the previous subsample library pointer is loaded and restored to continue training.

[0158] See Figure 11 After acquiring the training data used in the current batch, the computing device also records the training pointer and iteration steps. The assembly pointers of each DP node and the data processing process are independent of each other and do not interfere with each other. A pointer maintenance module is introduced to record the pointer of each subsample library in each iteration. Communication only occurs when saving ckpt / loading ckpt for continued training. While maintaining high training efficiency, it supports breakpoint resume training.

[0159] Optionally, the computing device can also save the identification information at the current moment when the number of iterations in the current training step reaches the save limit. The identification information is used to identify unused training data at the current moment and to resume training after the model training is interrupted. The identification information can be any kind of information that can be used to uniquely identify training data, including but not limited to the index and key of the training data.

[0160] The identification information can be either used or unused training data. For identification information indicating used training data, the computing device can store all used identification information or the last used identification information at the current moment. By storing the used training data identification information, the computing device can use the next training data point indicated by the identification information as the training data to be used when resuming training after a model interruption. For identification information indicating unused training data, the computing device can store all unused identification information or the first of multiple unused identification information stored at the current moment.

[0161] In one possible scenario, the model training method provided in this application is applicable to parallel training scenarios and can achieve parallel training through arbitrary computing resources. These computing resources include data loaders and DP nodes. DP nodes are associated with artificial intelligence (AI) computing power, and data loaders are associated with central processing units (CPUs). However, since computing resources on large-scale clusters also change in real time, in the event of a model training interruption and retraining, computing resources may increase or decrease. The computing device can dynamically adjust the training data indicated by the training pointer based on changes in computing resources after saving the training pointer.

[0162] Optionally, if model training is performed based on at least one data process and at least one DP node, the computing device can adjust the unused training data in at least one dimension of the DP node or data process based on the changed number of nodes or processes, and continue training using the adjusted training data, even if at least one of the number of DP nodes or the number of data processes changes. The change in quantity can be either an increase or a decrease; the process of adjusting the training data in different cases will be explained below.

[0163] Scenario 1: The number of nodes increases.

[0164] Optionally, if the number of nodes needs to be increased N times, the computing device can split the multiple training data points corresponding to each DP node into N parts, thereby adjusting the dimension of the multiple training data points at the DP node. Here, N is any positive integer greater than 1, and the computing device can sequentially split the multiple training data points according to their index order. Continuing with... Figure 15Taking the black-marked subsample library corresponding to the DP node with DP level = 0 in (1) as an example, the computing device can allocate multiple sets of training data included in the subsample library to subsample library 1 and subsample library 2. For example, the first set of training data is allocated to subsample library 1, the second set of training data is allocated to subsample library 2, the third set of training data is allocated to subsample library 1, and so on, to complete the sequential allocation of multiple sets of training data to different subsample libraries based on the index order.

[0165] Because the training data, originally located in the same subsample corpus, has been split into different subsample corpora, the position of the training data within each subsample corpus has changed. The computing device can generate new data pointers in the newly created subsample corpora to indicate the changed position of the training data. Based on a similar principle, in addition to adjusting the data pointers, the computing device also adjusts the training pointers, for example, by dividing the saved training pointers by N. Continuing... Figure 15 For example, the computing device will also divide the stored training pointers by 2, Figure 15 Divide the training pointer (cursor) of the subsample library with a value of 300 in (1) by 2 to obtain the cursors of the two subsample libraries, both of which are 150.

[0166] Scenario 2: The number of processes increases.

[0167] Similar to the operation in Case 1, if the number of processes needs to be increased to N times the original number, the multiple sets of training data corresponding to each data process can be split into N groups. During the splitting process, the original index order of the multiple sets of training data is preserved, the training pointers are also divided by N, and new data pointers are generated in the N split sub-sample databases. Continuing with... Figure 15 Taking (1) as an example of the sample database before splitting, the number of processes in the data process changes from 2 to 4, that is, the N-fold expansion is 2 times. The sample database after splitting is Figure 15 (3)

[0168] Scenario 3: The number of nodes decreases.

[0169] Optionally, if it is necessary to reduce the number of nodes to 1 / N of the original, the computing device can merge multiple sets of training data corresponding to multiple DP nodes, thereby adjusting the dimensionality of the multiple sets of training data at the DP nodes. Similar to the splitting process, the computing device can also merge the training data sequentially according to the index order of multiple sets of training data. See also Figure 16 In (1) and (2), the number of DP nodes is reduced to half of its original number, that is, the number of nodes changes from 2 to 1. In this case, the computing device can merge the subsample libraries corresponding to the two DP nodes into one subsample library, thereby adjusting the dimension of the training data in the DP nodes. The merging process will also be carried out sequentially according to the index order of multiple sets of training data in the subsample library, so as to... Figure 16Taking the black-marked subsample library 1 and the diagonally marked subsample library 2 in (1) as examples, the first set of training data in subsample library 1 is used as the first set of training data in the merged subsample library, the first set of training data in subsample library 2 is used as the second set of training data in the merged subsample library, the second set of training data in subsample library 1 is used as the third set of training data in the merged subsample library, and so on, to complete the sequential merging of multiple sets of training data into the same subsample library based on the index order.

[0170] Similar to the operation in Case 1, after merging multiple sets of training data, the computing device will also generate data pointers for each set of training data in the merged subsample library, and add the training pointers of the subsample library to obtain the training pointer of the merged subsample library.

[0171] Scenario 4: The number of processes decreases.

[0172] Similar to the operation process in scenario three, if it is necessary to reduce the number of processes to 1 / N of the original, the multiple sets of training data corresponding to the N data processes can be merged into one. During the merging process, the original index order of the multiple sets of training data is preserved, and a new pointer is generated in the merged subsample library. The training pointers corresponding to the N data processes are added together to obtain the training pointer of the merged subsample library. Continuing with... Figure 16 Taking (1) as an example of the sample database before the merger, the number of processes in the data process changes from 2 to 1, that is, the reduction of 1 / N is 1 / 2. See the sample database after the merger. Figure 16 (3)

[0173] Optionally, the computing device can, based on changes in the number of data processes and DP nodes, execute the adjustment process corresponding to any of the cases one through four, adjusting the training data in either the data process dimension or the DP node dimension. The computing device can also adjust the training data in multiple dimensions, such as... Figure 15 As shown, when the number of DP nodes and the number of data processes are doubled, the multiple sets of training data in the sample library are split into... Figure 15 The training data comes from the 16 subsample libraries in (4). The computing device can also be like... Figure 16 As shown, with the number of DP nodes and the number of data processes reduced by half, multiple training data sets from the four sub-sample databases are merged into one. Figure 16 The training data in one subsample library of (4).

[0174] Furthermore, the computing device can perform splitting and merging, for example, splitting at the DP node level and merging at the data process level, or merging at the DP node level and splitting at the data process level. By splitting or merging the capacity and pointers of the sub-sample library, it is possible to support expansion / shrinking during breakpoint training without affecting the order of subsequent data retrieval or the capabilities of the first trained model, thereby improving the competitiveness of model training in clustered deployments.

[0175] For example, if the computing device expands / shrinks during breakpoint training and saves identification information to identify unused training data, the identification information saved by the computing device includes either the identification information of all unused training data or the identification information of all used training data. By saving the identification information of each piece of unused or used training data, it is ensured that after adjustments such as expansion / shrinkage, the unused and used training data in the adjusted training data can be clearly identified, resulting in high reliability of training data indication.

[0176] Optionally, after training the first model, the computing device can perform multimodal processing on the first data using the first model to obtain the multimodal processing result, and output the second data based on the multimodal processing result. Next, we will take the multimodal processing task performed by the first model as visual question answering, with the first data including images and text, as an example to illustrate the process of the computing device performing multimodal processing. Visual question answering refers to understanding the content of an image and answering the question using natural language, given an image and a question. The question might be, for example, what is happening in the image, what color are the clothes the person is wearing, or how many dogs are in the image. In this case, the image in the first data is the image to be understood, and the text in the first data is the question to be answered.

[0177] The computing device understands the content of the image in the first data using a first model, determines the image content corresponding to the question, determines the multimodal processing result based on the image content, and generates text data as the answer based on the multimodal processing result. This text data is then used as the output second data. For example, in the case where the question is "What color is the person's clothing?", the computing device understands the image content and determines that the image content corresponding to the question is clothing. Based on the fact that the clothing color is black, the multimodal processing result is determined to be "The clothes are black," and the output text is "Black," which is the second data.

[0178] In summary, adjusting the first reference difference between the training data (first training data and second training data) occurs after the training data is loaded. That is, the first model obtained in this application is adjusted online during model training based on the training data. The training data being adjusted is the already loaded training data, not all the training data in the training dataset. This results in a smaller amount of training data to be adjusted and higher balancing efficiency. By reducing the first reference difference between the first and second training data, the length dispersion of the training data used for model training is reduced, leading to higher efficiency in training the model using training data with smaller length differences.

[0179] This application does not limit the adjustment of the first reference difference; it can be the first reference difference corresponding to the data length, thereby improving the load balancing of different DP nodes and reducing the bubbling rate. The computing device can also adjust the first reference difference corresponding to the first length, thereby improving the efficiency of different DP nodes encoding non-text data. Furthermore, the adjustment of the first reference difference corresponding to the first length can be achieved by moving non-text data, without requiring a global traversal of the dataset, thus avoiding startup time consumption, preventing disruption of the sampling order, maintaining training stability, and being suitable for various modalities or encoder network structures. It is perfectly compatible with dynamic data assembly and has high usability. Moreover, after adjusting the first reference difference corresponding to the data length, the total number of iterations is re-acquired, preventing overfitting or learning rate curves that do not match reality due to inaccurate total iterations during model training, thereby ensuring the model performance of the first trained model, compared to... Figure 10 The model shown is trained using training data assembled offline with relevant techniques. Its performance is not affected; for example, the model converges without loss, but performance metrics such as training throughput and model flops utilization (MFU) are improved.

[0180] The above describes the multimodal processing method of this application embodiment. Corresponding to the above method, this application embodiment also provides a multimodal processing device. Figure 17 This is a schematic diagram of the structure of a multimodal processing device provided in an embodiment of this application. Based on Figure 17 The following modules are shown. Figure 17 The multimodal processing device shown is capable of performing the above-mentioned functions. Figure 5 The illustrated computing device can operate in whole or in part. It should be understood that the device may include more additional modules than those shown, or may omit some of the modules shown; this application does not limit this. Figure 17 As shown, the device includes:

[0181] Module 1701 is used to acquire the first data to be processed;

[0182] The processing module 1702 is used to perform multimodal processing on the first data through the first model to obtain the multimodal processing result. The first model is trained using the third training data and the fourth training data.

[0183] The third and fourth training data are obtained by adjusting the first reference difference between the loaded first and second training data. The first reference difference indicates the length dispersion of the first and second training data. The second reference difference between the third and fourth training data is less than the first reference difference between the first and second training data.

[0184] In one possible implementation, the first reference difference between the first training data and the second training data includes at least one of the following: the first reference difference is the reference difference between the data length of the first training data and the data length of the second training data; or, the first reference difference is the reference difference between the first length corresponding to the first training data and the first length corresponding to the second training data, wherein the first length corresponding to the first training data indicates the length of non-text type data in the first training data; or, the first length corresponding to the first training data indicates the length of non-text type data in the first training data and the fifth training data, wherein the first length corresponding to the second training data indicates the length of non-text type data in the second training data; or, the first length corresponding to the second training data indicates the length of non-text type data in the second training data and the sixth training data.

[0185] In one possible implementation, the third training data is obtained based on the concatenated first training data, and the fourth training data is obtained based on the concatenated second training data; the concatenated first training data is obtained based on the fifth training data and the first training data, and the concatenated second training data is obtained based on the sixth training data and the second training data; the data length of the concatenated first training data is equal to the sum of the data lengths of the fifth training data and the first training data, and the data length of the concatenated second training data is equal to the sum of the data lengths of the sixth training data and the second training data; the second reference difference between the data lengths of the concatenated first training data and the data lengths of the concatenated second training data is less than the first reference difference.

[0186] In one possible implementation, the fifth training data is obtained from a selected data pile based on the type of the first training data. The multiple data piles are divided based on the type of training data, and any data pile includes at least one loaded training data.

[0187] In one possible implementation, the third training data and the first training data correspond to the same DP node, the fourth training data and the second training data correspond to the same DP node, the first length corresponding to the first training data is less than the first length corresponding to the second training data, the fourth training data is obtained by moving selected non-text data from the non-text data on the DP node corresponding to the second training data, the third training data is obtained by moving the selected non-text data to the DP node corresponding to the first training data, and the second reference difference between the first length corresponding to the fourth training data and the first length corresponding to the third training data is less than the first reference difference.

[0188] In one possible implementation, there is a training interruption during the training of the first model. The first model is then trained again based on the training pointer or identification information. The training pointer or identification information is saved based on the number of iterative training steps during the model training process. The training pointer or identification information is used to identify unused training data.

[0189] In one possible implementation, model training is performed based on at least one data process and at least one DP node. If at least one of the number of DP nodes or the number of data processes changes, the unused training data is adjusted based on at least one of the changed number of nodes or processes in at least one dimension of the DP nodes or data processes.

[0190] In one possible implementation, the first reference difference includes the reference difference between the data length of the first training data and the data length of the second training data. The first model is obtained by iterative training using the third and fourth training data based on the total number of iterations. The total number of iterations indicates the number of times the first model was obtained through iterative training.

[0191] The aforementioned device adjusts the first reference difference between the training data (first training data and second training data) after the training data is loaded. That is, this application adjusts the difference online during model training based on the training data. The training data being adjusted is the already loaded training data, not all the training data in the training dataset. The amount of training data to be adjusted is small, resulting in high balancing efficiency. By reducing the first reference difference between the first and second training data, the length dispersion of the training data used for model training is reduced, i.e., the length difference of the training data. Using training data with smaller length differences results in higher efficiency for model training.

[0192] It should be understood that the above Figure 17The provided device, in implementing its functions, is only illustrated by the division of the aforementioned functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here. Figure 17 The provided device can be a chip or a communication device, for example, Figure 17 The provided device may be one of the following network devices.

[0193] See Figure 18 , Figure 18 A schematic diagram of the structure of a network device 2000 provided in an exemplary embodiment of this application is shown. Figure 18 The network device 2000 shown is used to perform the above. Figure 5 The operations involved in the multimodal processing method shown are illustrated. The network device 2000 is, for example, a switch, a router, etc., and can be implemented using a general bus architecture.

[0194] like Figure 18 As shown, the network device 2000 includes at least one processor 2001, a memory 2003, and at least one communication interface 2004.

[0195] Processor 2001 may be, for example, a general-purpose central processing unit (CPU), a digital signal processor (DSP), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the solutions of this application. For example, processor 2001 includes application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A PLD may be, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof. It can implement or execute the various logic blocks, modules, and circuits described in connection with the embodiments of this application. A processor may also be a combination that implements computational functions, such as including one or more microprocessor combinations, a combination of a DSP and a microprocessor, etc.

[0196] Optionally, the network device 2000 also includes a bus. The bus is used to transfer information between the various components of the network device 2000. The bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 18 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0197] Memory 2003 may be, for example, read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; electrically erasable programmable read-only memory (EEPROM); compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.); magnetic disk storage media or other magnetic storage devices; or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 2003 may exist independently and be connected to processor 2001 via a bus. Memory 2003 may also be integrated with processor 2001.

[0198] The communication interface 2004 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), or Wireless Local Area Network (WLAN). The communication interface 2004 can include wired and wireless communication interfaces. Specifically, the communication interface 2004 can be an Ethernet interface, a Fast Ethernet (FE) interface, a Gigabit Ethernet (GE) interface, an Asynchronous Transfer Mode (ATM) interface, a WLAN interface, a cellular network communication interface, or a combination thereof. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. In this embodiment, the communication interface 2004 can be used by the network device 2000 to communicate with other devices.

[0199] In a specific implementation, as one example, the processor 2001 may include one or more CPUs, such as... Figure 18 The CPU0 and CPU1 shown are examples of processors. Each of these processors can be a single-core processor or a multi-core processor. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0200] In a specific implementation, as one example, the network device 2000 may include multiple processors, such as... Figure 18 The processors shown are 2001 and 2005. Each of these processors can be a single-core processor or a multi-core processor. Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0201] In a specific implementation, as one example, the network device 2000 may further include output devices and input devices. The output device communicates with the processor 2001 and can display information in various ways. For example, the output device may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device communicates with the processor 2001 and can receive user input in various ways. For example, the input device may be a mouse, keyboard, touchscreen device, or sensor device, etc.

[0202] In some embodiments, the memory 2003 stores program code 2010 for executing the solution of this application, and the processor 2001 can execute the program code 2010 stored in the memory 2003. That is, the network device 2000 can implement the model training method or multimodal processing method provided in the method embodiments through the processor 2001 and the program code 2010 in the memory 2003. The program code 2010 may include one or more software modules. Optionally, the processor 2001 itself may also store program code or instructions for executing the solution of this application.

[0203] In a specific embodiment, the network device 2000 of this application embodiment may correspond to the computing device in the above-described method embodiments.

[0204] in, Figure 5Each step of the multimodal processing method shown is completed through integrated logic circuits in the hardware or software instructions in the processor of the network device 2000. The steps of the method disclosed in the embodiments of this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. Since this storage medium is located in memory, the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method; to avoid repetition, these will not be described in detail here.

[0205] See Figure 19 , Figure 19 This invention provides a schematic diagram of the structure of a network device 2100 according to another exemplary embodiment of the present application. Figure 19 The network device 2100 shown is used to perform the above. Figure 5 The multimodal processing method shown involves all or part of the operations. The network device 2100 is, for example, a switch, a router, etc., and can be implemented using a general bus architecture.

[0206] like Figure 19 As shown, the network device 2100 includes a main control board 2110 and an interface board 2130.

[0207] The main control board, also known as the main processing unit (MPU) or route processor card, is used to control and manage the various components in the network device 2100, including routing calculation, device management, device maintenance, and protocol processing functions. The main control board 2110 includes a central processing unit 2111 and a memory 2112.

[0208] Interface board 2130 is also called a line processing unit (LPU), linecard, or service board. Interface board 2130 provides various service interfaces and implements packet forwarding. Service interfaces include, but are not limited to, Ethernet interfaces, POS (Packet over SONET / SDH) interfaces, etc., with Ethernet interfaces including, for example, Flexible Ethernet Clients (FlexE Clients). Interface board 2130 includes: a central processing unit 2131, a network processor 2132, a forwarding table entry memory 2134, and a physical interface card (PIC) 2133.

[0209] The central processing unit 2131 on the interface board 2130 is used to control and manage the interface board 2130 and communicate with the central processing unit 2111 on the main control board 2110.

[0210] Network processor 2132 is used to implement packet forwarding processing. Network processor 2132 can be in the form of a forwarding chip. The forwarding chip can be a network processor (NP). In some embodiments, the forwarding chip can be implemented using an application-specific integrated circuit (ASIC) or a field-programmable gate array (FPGA). Specifically, network processor 2132 forwards received packets based on the forwarding table stored in forwarding table entry memory 2134. If the destination address of the packet is the address of network device 2100, the packet is sent to the CPU (such as central processing unit 2131) for processing; if the destination address of the packet is not the address of network device 2100, the next hop and outgoing interface corresponding to the destination address are looked up in the forwarding table according to the destination address, and the packet is forwarded to the outgoing interface corresponding to the destination address. Uplink packet processing may include: packet ingress interface processing, forwarding table lookup; downlink packet processing may include: forwarding table lookup, etc. In some embodiments, the central processing unit can also perform the functions of the forwarding chip, such as implementing software forwarding based on a general-purpose CPU, thus eliminating the need for a forwarding chip on the interface board.

[0211] The physical interface card 2133 is used to implement physical layer interfacing functions. Raw traffic enters the interface board 2130 through this card, and processed packets are sent out from the physical interface card 2133. The physical interface card 2133, also called a daughter card, can be installed on the interface board 2130. It is responsible for converting photoelectric signals into packets, performing validity checks on the packets, and forwarding them to the network processor 2132 for processing. In some embodiments, the central processing unit 2131 can also perform the functions of the network processor 2132, such as implementing software forwarding based on a general-purpose CPU, thus eliminating the need for the network processor 2132 in the physical interface card 2133.

[0212] Optionally, network device 2100 includes multiple interface boards. For example, network device 2100 also includes interface board 2140, which includes: a central processing unit 2141, a network processor 2142, a forwarding table entry memory 2144, and a physical interface card 2143. The functions and implementation methods of each component in interface board 2140 are the same as or similar to those in interface board 2130, and will not be described in detail here.

[0213] Optionally, network device 2100 also includes a switching fabric board 2120. The switching fabric board 2120 can also be referred to as a switch fabric unit (SFU). When network device 2100 has multiple interface boards, the switching fabric board 2120 is used to complete data exchange between the interface boards. For example, interface boards 2130 and 2140 can communicate through the switching fabric board 2120.

[0214] The main control board 2110 and the interface boards are coupled. For example, the main control board 2110, interface boards 2130 and 2140, and the switching network board 2120 communicate with each other via a system bus connected to the system backplane. In one possible implementation, an inter-process communication (IPC) channel is established between the main control board 2110 and interface boards 2130 and 2140, and communication between the main control board 2110 and interface boards 2130 and 2140 is achieved through the IPC channel.

[0215] Logically, network device 2100 includes a control plane and a forwarding plane. The control plane includes a main control board 2110 and a central processing unit 2111, while the forwarding plane includes various components that perform forwarding, such as a forwarding table entry memory 2134, a physical interface card 2133, and a network processor 2132. The control plane performs functions such as router operation, generating forwarding tables, processing signaling and protocol messages, and configuring and maintaining the status of network devices. The control plane distributes the generated forwarding tables to the forwarding plane. In the forwarding plane, the network processor 2132 forwards messages received by the physical interface card 2133 based on the forwarding tables distributed by the control plane. The forwarding tables distributed by the control plane can be stored in the forwarding table entry memory 2134. In some embodiments, the control plane and the forwarding plane can be completely separated and not on the same network device.

[0216] It's worth noting that a network device may have one or more main control boards, including a primary and a backup main control board. It may also have one or more interface boards; the more powerful the network device's data processing capabilities, the more interface boards it provides. Each interface board may also have one or more physical interface cards. A switching board may or may not exist; multiple boards can share the load and provide redundancy. In a centralized forwarding architecture, the network device may not need a switching board, as the interface boards handle the entire system's business data processing. In a distributed forwarding architecture, the network device can have at least one switching board, which enables data exchange between multiple interface boards, providing high-capacity data exchange and processing capabilities. Therefore, the data access and processing capabilities of a distributed architecture network device are greater than those of a centralized architecture network device. Alternatively, the network device can also be a single board, without a switching board. The functions of the interface board and the main control board are integrated on this one board. In this case, the central processing unit (CPU) on the interface board and the CPU on the main control board can be combined into a single CPU to perform the combined functions. This type of network device has lower data exchange and processing capabilities (e.g., low-end switches or routers). The specific architecture adopted depends on the specific network deployment scenario, and no restrictions are imposed here.

[0217] In a specific embodiment, network device 2100 corresponds to the above. Figure 17 The apparatus shown. In some embodiments, Figure 17 The processing module 1702 in the multimodal processing device shown is equivalent to the central processing unit 2111 or network processor 2132 in the network device 2100.

[0218] This application embodiment also provides a communication device, which includes a transceiver, a memory, and a processor. The transceiver, the memory, and the processor communicate with each other via an internal connection path. The memory stores instructions, and the processor executes the instructions stored in the memory to control the transceiver to receive and transmit signals. When the processor executes the instructions stored in the memory, it causes the processor to perform... Figure 5 The multimodal processing method shown.

[0219] It should be understood that the aforementioned processor can be a CPU, or other general-purpose processors, DSPs, ASICs, FPGAs, or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting the Advanced Reduced Instruction Set Computing (RISC) machine (ARM) architecture.

[0220] Furthermore, in an alternative embodiment, the memory described above may include read-only memory and random access memory, and provide instructions and data to the processor. The memory may also include non-volatile random access memory. For example, the memory may also store device type information.

[0221] The memory can be volatile or non-volatile, or may include both. Non-volatile memory can be ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), EEPROM, or flash memory. Volatile memory can be RAM, used as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0222] This application embodiment also provides a computer device, the device including a processor, the processor being configured to load and execute at least one instruction to cause the computer device to perform, as in... Figure 5 The multimodal processing method is illustrated. Optionally, the device further includes a memory coupled to the processor, the memory being used to store at least one instruction.

[0223] This application also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to enable the computer to perform the following: Figure 5 The multimodal processing method shown is illustrated.

[0224] This application also provides a computer program (product) that, when executed by a computer, causes the processor or computer to perform the corresponding steps and / or processes in the above method embodiments.

[0225] This application also provides a chip, which includes a processor for calling and executing instructions stored in a memory, causing a communication device equipped with the chip to perform actions such as... Figure 5 The multimodal processing method shown.

[0226] This application embodiment also provides another chip, including: an input interface, an output interface, a processor, and a memory. The input interface, the output interface, the processor, and the memory are connected via internal interconnection paths. The processor is used to execute code in the memory. When the code is executed, the processor is used to perform actions such as... Figure 5 The multimodal processing method shown.

[0227] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive).

[0228] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the first training data and the second training data involved in this application were obtained with full authorization.

[0229] Those skilled in the art will recognize that the method steps and modules described in conjunction with the embodiments disclosed herein can be implemented in software, hardware, firmware, or any combination thereof. To clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0230] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0231] When implemented using software, it can be implemented wholly or partially as a computer program product. This computer program product includes one or more computer program instructions. As an example, the methods of this application embodiment can be described in the context of machine-executable instructions, such as program modules that execute on a device on a real or virtual processor of the target. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., which perform specific tasks or implement specific abstract data structures. In various embodiments, the functionality of program modules can be combined or divided among the described program modules. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside on both local and remote storage media.

[0232] Computer program code used to implement the methods of the embodiments of this application may be written in one or more programming languages. This computer program code may be provided to the processor of a general-purpose computer, a special-purpose computer, or other programmable model training device or multimodal processing device, such that when executed by the computer or other programmable model training device or multimodal processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0233] In the context of the embodiments of this application, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, etc.

[0234] Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0235] A machine-readable medium can be any tangible medium that contains or stores programs for or relating to an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More detailed examples of machine-readable storage media include electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0236] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be found in the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0237] In the embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or modules, or they may be electrical, mechanical, or other forms of connection.

[0238] The modules described as separate components may or may not be physically separate. Similarly, the components shown as modules may or may not be physical modules; they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0239] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0240] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0241] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have substantially the same function and purpose. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the various examples described, a first image can be referred to as a second image, and similarly, a second image can be referred to as a first image. Both the first image and the second image can be images, and in some cases, they can be separate and distinct images.

[0242] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0243] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple second messages refer to two or more second messages. The terms "system" and "network" are often used interchangeably in this document.

[0244] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0245] It should also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" describes an association between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects are in an "or" relationship.

[0246] It should also be understood that the term “comprising” (also referred to as “includes”, “including”, “comprises” and / or “comprising”) as used in this specification specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0247] It should also be understood that the terms “if” and “if” can be interpreted as meaning “when” or “upon”, or “in response to determination” or “in response to detection”. Similarly, depending on the context, the phrases “if determination…” or “if detection [the stated condition or event]” can be interpreted as meaning “when determination…”, or “in response to determination…”, or “when detection [the stated condition or event]” or “in response to detection [the stated condition or event]”.

[0248] It should be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0249] It should also be understood that the phrases "an embodiment," "an embodiment," and "a possible implementation" used throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment or implementation is included in at least one embodiment of this application. Therefore, the phrases "in an embodiment," "an embodiment," or "a possible implementation" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

Claims

1. A multimodal processing method, characterized in that, The method includes: Get the first data to be processed; The first model performs multimodal processing on the first data to obtain the multimodal processing result. The first model is trained using the third and fourth training data. The third training data and the fourth training data are obtained by adjusting the first reference difference between the loaded first training data and the second training data. The first reference difference indicates the length dispersion of the first training data and the second training data. The second reference difference between the third training data and the fourth training data is smaller than the first reference difference between the first training data and the second training data.

2. The method according to claim 1, characterized in that, The first reference difference between the first training data and the second training data includes at least one of the following: The first reference difference is the reference difference between the data length of the first training data and the data length of the second training data; Alternatively, the first reference difference can be the reference difference between the first length corresponding to the first training data and the first length corresponding to the second training data, where the first length corresponding to the first training data indicates the length of non-text data in the first training data; or, the first length corresponding to the first training data indicates the length of non-text data in the first training data and the fifth training data, where the first length corresponding to the second training data indicates the length of non-text data in the second training data; or, the first length corresponding to the second training data indicates the length of non-text data in the second training data and the sixth training data.

3. The method according to claim 2, characterized in that, The third training data is obtained based on the spliced ​​first training data, and the fourth training data is obtained based on the spliced ​​second training data; The first training data after splicing is obtained based on the fifth training data and the first training data. The second training data after splicing is obtained based on the sixth training data and the second training data. The data length of the first training data after splicing is equal to the sum of the data lengths of the fifth training data and the first training data. The data length of the second training data after splicing is equal to the sum of the data lengths of the sixth training data and the second training data. The second reference difference between the data lengths of the first training data after splicing and the data lengths of the second training data after splicing is less than the first reference difference.

4. The method according to claim 2 or 3, characterized in that, The fifth training data is obtained from a selected data pile from a plurality of data piles based on the type of the first training data. The plurality of data piles are divided based on the type of training data, and each data pile includes at least one loaded training data.

5. The method according to any one of claims 2-4, characterized in that, The third training data and the first training data correspond to the same data parallel DP node, the fourth training data and the second training data correspond to the same DP node, and the first length corresponding to the first training data is less than the first length corresponding to the second training data. The fourth training data is obtained by moving selected non-text data from the non-text data on the DP node corresponding to the second training data. The third training data is obtained by moving the selected non-text data to the DP node corresponding to the first training data. The second reference difference between the first length corresponding to the fourth training data and the first length corresponding to the third training data is less than the first reference difference.

6. The method according to any one of claims 1-5, characterized in that, During the training of the first model, there may be a training interruption. The first model can be trained again after the training interruption based on the training pointer or the identification information. The training pointer or the identification information is saved based on the number of iterative training steps in the model training process. The training pointer or the identification information is used to identify unused training data.

7. The method according to claim 6, characterized in that, The model training is performed based on at least one data process and at least one data parallel DP node. If at least one of the number of DP nodes or the number of data processes changes, the unused training data is adjusted based on at least one of the changed number of nodes or processes in at least one dimension of the DP node or data process.

8. The method according to any one of claims 1-7, characterized in that, The first reference difference includes the reference difference between the data length of the first training data and the data length of the second training data. The first model is obtained by iterative training using the third training data and the fourth training data based on the total number of iterations. The total number of iterations indicates the number of times the first model was obtained through iterative training.

9. A multimodal processing device, characterized in that, The device includes: The acquisition module is used to acquire the first data to be processed; The processing module is used to perform multimodal processing on the first data using a first model to obtain multimodal processing results. The first model is trained using third and fourth training data. The third training data and the fourth training data are obtained by adjusting the first reference difference between the loaded first training data and the second training data. The first reference difference indicates the length dispersion of the first training data and the second training data. The second reference difference between the third training data and the fourth training data is smaller than the first reference difference between the first training data and the second training data.

10. The apparatus according to claim 9, characterized in that, The first reference difference between the first training data and the second training data includes at least one of the following: the first reference difference is the reference difference between the data length of the first training data and the data length of the second training data; or, the first reference difference is the reference difference between the first length corresponding to the first training data and the first length corresponding to the second training data, wherein the first length corresponding to the first training data indicates the length of non-text type data in the first training data; or, the first length corresponding to the first training data indicates the length of non-text type data in the first training data and the fifth training data, wherein the first length corresponding to the second training data indicates the length of non-text type data in the second training data; or, the first length corresponding to the second training data indicates the length of non-text type data in the second training data and the sixth training data.

11. The apparatus according to claim 10, characterized in that, The third training data is obtained based on the spliced ​​first training data, and the fourth training data is obtained based on the spliced ​​second training data; The first training data after splicing is obtained based on the fifth training data and the first training data. The second training data after splicing is obtained based on the sixth training data and the second training data. The data length of the first training data after splicing is equal to the sum of the data lengths of the fifth training data and the first training data. The data length of the second training data after splicing is equal to the sum of the data lengths of the sixth training data and the second training data. The second reference difference between the data lengths of the first training data after splicing and the data lengths of the second training data after splicing is less than the first reference difference.

12. The apparatus according to claim 10 or 11, characterized in that, The fifth training data is obtained from a selected data pile from a plurality of data piles based on the type of the first training data. The plurality of data piles are divided based on the type of training data, and each data pile includes at least one loaded training data.

13. The apparatus according to any one of claims 10-12, characterized in that, The third training data and the first training data correspond to the same data parallel DP node, the fourth training data and the second training data correspond to the same DP node, and the first length corresponding to the first training data is less than the first length corresponding to the second training data. The fourth training data is obtained by moving selected non-text data from the non-text data on the DP node corresponding to the second training data. The third training data is obtained by moving the selected non-text data to the DP node corresponding to the first training data. The second reference difference between the first length corresponding to the fourth training data and the first length corresponding to the third training data is less than the first reference difference.

14. The apparatus according to any one of claims 9-13, characterized in that, During the training of the first model, there may be a training interruption. The first model can be trained again after the training interruption based on the training pointer or the identification information. The training pointer or the identification information is saved based on the number of iterative training steps in the model training process. The training pointer or the identification information is used to identify unused training data.

15. The apparatus according to claim 14, characterized in that, The model training is performed based on at least one data process and at least one data parallel DP node. If at least one of the number of DP nodes or the number of data processes changes, the unused training data is adjusted based on at least one of the changed number of nodes or processes in at least one dimension of the DP node or data process.

16. The apparatus according to any one of claims 9-15, characterized in that, The first reference difference includes the reference difference between the data length of the first training data and the data length of the second training data. The first model is obtained by iterative training using the third training data and the fourth training data based on the total number of iterations. The total number of iterations indicates the number of times the first model was obtained through iterative training.

17. A computer device, characterized in that, The device includes a processor coupled to a memory storing at least one program instruction or code, which is loaded and executed by the processor to enable the computer device to implement the multimodal processing method as described in any one of claims 1-8.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement the multimodal processing method as described in any one of claims 1-8.

19. A chip, characterized in that, The chip includes a processor for running program instructions or code that cause a device containing the chip to perform the multimodal processing method as described in any one of claims 1-8.

20. A computer program product, characterized in that, The computer program product includes a computer program / instruction that is executed by a processor to cause a computer to perform the multimodal processing method as described in any one of claims 1-8.

Images