Model training multi-modal data combination method, device, equipment and medium

Abstract

This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and medium for combining multimodal data for model training. This method utilizes the differences in modality to group training data of the same modality into a single modality grouping result, ensuring that the data lengths within this grouping result are similar. The data is then sorted by length to obtain a neat sequence of data, which is then segmented according to the amount of data processed in a batch during training. Because the modality grouping and sorting conditions ensure that the data lengths in the first segmented data group are comparable and meet the data volume requirements for a batch, excessive padding is unnecessary, thus effectively reducing the waste of computational resources.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device and medium for combining multimodal data for model training. Background Technology

[0002] Currently, multimodal technology is developing rapidly and playing an increasingly important role in people's work and life. However, obtaining a multimodal model often requires a long training period. Many factors affect the efficiency of multimodal training, one of which is the combination of data. Typically, model training involves randomly shuffling the data and then using a sampler to combine batches of data. During combination, due to the varying lengths of the data, padding operations are required to form a matrix, thus enabling parallel training of samples. When training multimodal models, the data used covers various modalities, and the significant differences in data length between different modalities lead to more padding operations, resulting in a waste of computational resources.

[0003] Therefore, how to combine multimodal data used for model training to reduce the waste of computing resources caused by data length padding operations has become an urgent problem to be solved. Summary of the Invention

[0004] In view of this, embodiments of this application provide a method, apparatus, device, and medium for combining multimodal data for model training, in order to solve the problem of how to combine multimodal data for model training in order to reduce the waste of computing resources caused by data length padding operations.

[0005] In a first aspect, embodiments of this application provide a method for combining multimodal data for model training, including: Obtain training data for N modalities, group all training data according to modality, and obtain the modality grouping results for each modality, where N is an integer greater than 1; For any modality grouping result, sort all the training data in the modality grouping result in ascending or descending order of data length to obtain the sequence data corresponding to the modality grouping result; Using a preset batch of data, the sequence data is segmented to obtain at least one first data group corresponding to the modality grouping result, and each first data group includes training data corresponding to the preset batch of data. Iterate through all modality grouping results to obtain the first data group corresponding to each modality grouping result. Merge all the first data groups to obtain the merged data, which is used to train the model.

[0006] Secondly, embodiments of this application provide a multimodal data combination device for model training, comprising: The modality grouping module is used to acquire training data for N modalities, group all training data according to modality, and obtain the modality grouping result for each modality, where N is an integer greater than 1; The data sorting module is used to sort all training data in any modality grouping result in ascending or descending order of data length to obtain sequence data corresponding to the modality grouping result. The batch segmentation module is used to segment the sequence data using a preset batch data volume to obtain at least one first data group corresponding to the modality grouping result, wherein each first data group includes training data corresponding to the preset batch data volume. The data combination module is used to traverse all modality grouping results, obtain the first data group corresponding to each modality grouping result, and combine all the first data groups to obtain the merged data, which is used to train the model.

[0007] Thirdly, embodiments of this application provide a computer device, the computer device including a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, it implements the multimodal data combination method for model training as described in the first aspect.

[0008] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multimodal data combination method for model training as described in the first aspect.

[0009] The beneficial effects of the embodiments in this application compared with the prior art are: The method of this application acquires training data of N modalities, groups all training data according to modality to obtain modality grouping results for each modality, sorts all training data in the modality grouping results according to the data length in ascending or descending order to obtain sequence data of the corresponding modality grouping results, uses a preset batch of data to segment the sequence data to obtain at least one first data group of the corresponding modality grouping results, each first data group includes training data of the preset batch of data, iterates through all modality grouping results to obtain the first data group corresponding to each modality grouping result, merges all first data groups to obtain merged data, and uses the merged data to train the model.

[0010] By utilizing different modalities, training data under the same modality are grouped into a single modal grouping result, making the data lengths in this modal grouping result similar. Then, the data is sorted by length to obtain a neat sequence of data, which is then split according to the amount of data processed in a batch during training. Because of the modal grouping and sorting conditions, the data lengths in the first data group obtained by splitting are similar and meet the data volume requirements for a batch processing, so there is no need for excessive padding, which can effectively reduce the waste of computing resources. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of an application environment for a multimodal data combination method for model training provided in Embodiment 1 of this application; Figure 2 This is a flowchart illustrating a method for combining multimodal data for model training provided in Embodiment 2 of this application; Figure 3 This is a flowchart illustrating a method for combining multimodal data for model training provided in Embodiment 3 of this application; Figure 4 This is a flowchart illustrating a method for combining multimodal data for model training provided in Embodiment 4 of this application; Figure 5 This is a schematic diagram of the structure of a multimodal data combination device for model training provided in Embodiment 5 of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided in Embodiment Six of this application. Detailed Implementation

[0013] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0014] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0015] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0016] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0017] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0018] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0019] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0020] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0021] It should be understood that the sequence number of each step in the following embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0022] To illustrate the technical solution of this application, specific embodiments are described below.

[0023] The multimodal data combination method for model training provided in Embodiment 1 of this application can be applied to, for example... Figure 1 In this application environment, the client communicates with the server. The client can send training data to the server, thereby enabling the server to train the model. Before training, the multimodal training data is preprocessed using the aforementioned model training multimodal data combination method.

[0024] This multimodal data combination method for model training can be applied to the server side, where the server deploys the model to be trained. After preprocessing the training data, the preprocessed training data is sent to the model for training. Alternatively, this method can also be applied to the client side, where the client obtains the training data, processes it, and then sends it to the server for model training. The client can correspond to a database, meaning the server directly retrieves the training data from the database.

[0025] The client side includes, but is not limited to, PDAs, desktop computers, laptops, ultra-mobile personal computers (UMPCs), netbooks, cloud terminal devices, and personal digital assistants (PDAs). The server side can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms.

[0026] See Figure 2This is a flowchart illustrating a method for combining multimodal data for model training provided in Embodiment 2 of this application. The aforementioned method for combining multimodal data for model training can be applied to... Figure 1 The server-side component connects to the client, database, and other resources to obtain and process the relevant data.

[0027] like Figure 2 As shown, the method for combining multimodal data for model training may include the following steps: Step S201: Obtain training data for N modalities, group all training data according to modality, and obtain the modality grouping results for each modality.

[0028] Where N is an integer greater than 1. Multimodal refers to data that has multiple different representations. For example, "an apple" can be represented as text or as an image containing an apple, and data that combines both is multimodal data. Multimodal data generally includes interactions between two modalities. For example, an image can be expressed through interactive text as "[Question]: What is in the picture? [Response]: There is an apple in the picture." Therefore, in a training sample of multimodal data, the image is the corresponding image information, and the text is the text interaction for that image.

[0029] Multimodal training is generally based on pre-trained large language models. Therefore, in actual training, a large amount of text data is added to maintain the model's language capabilities. This raises two issues that affect training efficiency. For example, text data and image-text data have significantly different lengths. For instance, text data might be x in length, while image-text data, due to the presence of images, has a length of 2x (text encoding length + image encoding length). When a batch of training data mixes pure text and image-text data, the data in the batch needs to be padded to a length of 2x, resulting in many useless tokens in the text data and increased ineffective computation. Furthermore, image-text data also involves encoding merging operations. When a batch contains text data, the merging operation cannot be performed in parallel, which also affects training efficiency.

[0030] In this embodiment, for the training data, all training data are grouped according to the modality of each training data, and training data under the same modality are grouped together. That is, each modality corresponds to a modality grouping result, and the modality grouping result includes all training data under that modality.

[0031] Modal grouping allows training data with the same modality to be grouped together, ensuring that the data lengths of the training data in the modal grouping result are all at the same level. Of course, the modal grouping result does not limit the existence of two training data sets with significantly different data lengths. Modal grouping can achieve a preliminary screening of data lengths for subsequent use.

[0032] Step S202: For any modality grouping result, sort all the training data in the modality grouping result according to the data length in ascending or descending order to obtain the sequence data of the corresponding modality grouping result.

[0033] In this embodiment, each modality grouping result is processed separately. In a modality grouping result, all training data are sorted, either by decreasing or increasing data length, to form serialized sequence data.

[0034] In this sequence of data, the length difference between two adjacent training data points is not significant. Therefore, when subsequent batches of training data are generated, the length difference within each batch will also be minimal. Of course, there may be abrupt changes in the length of adjacent training data points; filtering or other methods can be used to optimize for such abrupt changes.

[0035] Step S203: Using a preset batch of data, the sequence data is segmented to obtain at least one first data group of the corresponding modality grouping result.

[0036] Each of the first data groups includes training data of a pre-defined batch size. The pre-defined batch size is a value determined based on the model training requirements and the training environment. For example, when batch=5, only 5 training data points can be processed in a single batch of training.

[0037] In this embodiment, the sequence data is segmented in order to obtain training data for each batch. That is, after segmentation, at least one first data group is obtained. The amount of training data in a first data group is the preset amount of data for one batch. After segmentation, subsequent data processing needs to be performed on a unit basis of the first data group.

[0038] Segmentation can be performed according to the length of the data in the sequence data, from largest to smallest or smallest to largest, to obtain the segmented data groups. In a data group, since the training data is arranged by data length and is adjacent, the data lengths of the training data in a data group are similar, thus avoiding a lot of padding in subsequent processing.

[0039] Step S204: Traverse all modal grouping results to obtain the first data group corresponding to each modal grouping result, and merge all the first data groups to obtain the merged data.

[0040] The merged data is used to train the model. The merged data can be sequentially accessed and fed to the training model in the manner of the first data group to perform batch training on the model. The trained model includes, but is not limited to, neural network models, machine learning models, etc.

[0041] In this embodiment, steps S202 and S203 are performed on all modality grouping results respectively, thereby obtaining the first data group corresponding to each modality grouping result. All the first data groups are combined to form the training set composed of training data. The data merging can be done by arranging all the first data groups in sequence to form a one-dimensional matrix or a two-dimensional matrix, etc. Of course, combined with the dimension of the training data, it can also form a three-dimensional matrix of data.

[0042] This application embodiment acquires training data for N modalities, groups all training data according to modality to obtain modality grouping results for each modality, sorts all training data in the modality grouping results according to increasing or decreasing data length to obtain sequence data of the corresponding modality grouping results, uses a preset batch of data to segment the sequence data to obtain at least one first data group of the corresponding modality grouping results, each first data group includes training data corresponding to the preset batch of data, traverses all modality grouping results to obtain the first data group corresponding to each modality grouping result, merges all first data groups to obtain merged data, and uses the merged data to train the model. By utilizing different modalities, training data under the same modality are grouped into a single modal grouping result, making the data lengths in this modal grouping result similar. Then, the data is sorted by length to obtain a neat sequence of data, which is then split according to the amount of data processed in a batch during training. Because of the modal grouping and sorting conditions, the data lengths in the first data group obtained by splitting are similar and meet the data volume requirements for a batch processing, so there is no need for excessive padding, which can effectively reduce the waste of computing resources.

[0043] See Figure 3 This is a flowchart illustrating a method for combining multimodal data for model training provided in Embodiment 3 of this application. Figure 3 As shown, after using a preset batch of data to segment the sequence data in step S203 above to obtain at least one first data group of the corresponding modality grouping result, the following steps may also be included: Step S301: Randomly arrange all the first data groups of the corresponding modal grouping results to obtain the rearranged first data group of the corresponding modal grouping results.

[0044] The step S204 above, which involves traversing all modal grouping results to obtain the first data group corresponding to each modal grouping result, and merging all the first data groups to obtain the merged data, may include the following steps: Step S302: Traverse all modal grouping results to obtain the first data group after rearrangement for each modal grouping result, and merge all rearranged first data groups to obtain the merged data.

[0045] In this embodiment, since the basic number of data to be scheduled is a batch, the data is first sorted by length and then grouped into batches, and then randomly arranged (i.e., shuffled). The purpose of shuffling is to avoid the model training pattern being too fixed. If shuffling is not performed, the data length of the model training data will become a single pattern of short to long or long to short, which is not conducive to model training. In order to maintain the order of data length within a batch after shuffling, shuffling operation needs to be performed on a batch basis.

[0046] For example, taking the sequence data [D1,D2,D3,D4,D5,D6,D7,D8] after sorting by data length and with batch=2 as an example, it can be represented as [[D1,D2],[D3,D4],[D5,D6],[D7,D8]] after splitting. Among them, [D1,D2], [D3,D4], [D5,D6], and [D7,D8] are all first data groups. After shuffling in batches, the rearranged first data groups may be [[D3,D4],[D7,D8],[D1,D2],[D5,D6]]. Of course, the rearranged first data groups after shuffling may also have other arrangements, but the arrangement of each first data group will not change.

[0047] After obtaining all rearranged first data groups, they are merged to obtain the merged data. For example, one rearranged first data group is [[D3,D4],[D7,D8],[D1,D2],[D5,D6]], and another rearranged first data group is [[D9,D10],[D11,D12]]. Merging the two rearranged first data groups yields the merged data [[D3,D4],[D7,D8],[D1,D2],[D5,D6],[D9,D10],[D11,D12]].

[0048] Optionally, using a preset batch size of data, the sequence data is segmented to obtain at least one first data group of the corresponding modality grouping results, including: Based on a preset batch of data, the sequence data is divided sequentially from front to back to obtain M first data groups, where M is an integer greater than zero.

[0049] In this method, the sequence data is segmented sequentially from beginning to end. This eliminates the need to count the number of data points in the sequence data or use methods such as copying or extraction to obtain the data. Operations can be performed directly on the sequence data.

[0050] If the sequence data is sorted from largest to smallest, the length of the training data in the first data group, which is split sequentially from front to back, decreases sequentially. If the sequence data is sorted from smallest to largest, the length of the training data in the first data group, which is split sequentially from front to back, increases sequentially.

[0051] In this embodiment, the first data group is used as the unit for random arrangement, so that the data used for model training is not increasing or decreasing. That is, the length of the training data of the model is no longer a single pattern of short to long or long to short, which helps to improve the applicability of model training.

[0052] See Figure 4 This is a flowchart illustrating a method for combining multimodal data for model training provided in Embodiment 4 of this application. Figure 4 As shown, after combining all the first data in step S204 to obtain the merged data, the following steps may be included: Step S401: Using the number of graphics cards in the model training server, the merged data is grouped to obtain a second data group corresponding to the number of graphics cards.

[0053] The performance of the server used to train the model determines how the data is allocated during model training. If the server can only use serial training, then the number of graphics cards can be considered as 1, and the training data can only be fed to the model sequentially for training.

[0054] If the server can use parallel training, this parallel training can be either distributed data parallelism or non-distributed data parallelism. That is, multiple parallel graphics cards can run simultaneously on a server, each with a model to be trained. The training data is fed to the corresponding model to be trained in parallel to achieve parallel training. Alternatively, graphics cards in multiple service nodes can be used in parallel to train the model.

[0055] Therefore, the number of graphics cards (i.e., world_size) represents the amount of parallel training. If there are many graphics cards, the merged data can be grouped and assigned to the corresponding data group for each graphics card.

[0056] If the merged data is not grouped, it will be distributed to multiple graphics cards for processing sequentially. For example, if the merged data is [[D3,D4],[D7,D8],[D1,D2],[D5,D6]], assuming world_size=2 and batch=2, then [D3,D4], [D7,D8] will be sequentially distributed to the two graphics cards. In this case, the first graphics card gets D3, the second graphics card gets D4, then the first graphics card gets D7, and so on. Finally, the data of the first graphics card will be [D3,D7], and the expected ordered arrangement of [D3,D4] will fail.

[0057] For example, if the merged data is [[D3,D4],[D7,D8],[D1,D2],[D5,D6]], all first data groups in the merged data are divided into world_size groups, where world_size=2. For example, [D3,D4] and [D1,D2] are divided into world_size_1 = ([D3,D4,D1,D2]), and [D7,D8] and [D5,D6] are divided into world_size_2 = ([D7,D8,D5,D6]). In this way, world_size_1 is sent to the first graphics card, and world_size_2 is sent to the second graphics card, which will not disrupt the expected orderly arrangement.

[0058] Step S402: Tile all the second data groups to obtain the tiled data as the target data.

[0059] The number of rows in the target data is the number of graphics cards, the number of columns in the target data is the preset batch size, and the target data is used to train the model.

[0060] In this embodiment, after the merged data is grouped, a second data group is sent to a graphics card for processing. For a training set, all the second data groups need to be represented in one dataset. Therefore, it is necessary to perform a tiling operation on all the second data groups to obtain the target data, which is the data used to train the model.

[0061] The dimension of the target data is the number of graphics cards × the preset batch size, i.e., world_size × batch. The columns of the target data correspond to the graphics cards, which can ensure the original data arrangement and achieve data allocation.

[0062] Optionally, after grouping the merged data according to the number of GPUs in the server used for model training to obtain a second data group corresponding to the number of GPUs, the following may also be included: Randomly arrange all the second data groups, taking the first data group as a unit, to obtain rearranged second data groups; Tile all the second data sets to obtain the target data, which includes: Tile all rearranged second data groups to obtain the tiled data as the target data.

[0063] The random arrangement (i.e., shuffle) here is to avoid the model training pattern being too fixed. If shuffle is not performed, the model training data may not be randomly distributed, which is not conducive to model training. In order to maintain the order of data length within a batch after shuffle, shuffle operation is required between different second data groups in batches.

[0064] Optionally, the merged data can be grouped according to the number of GPUs in the server used for model training, resulting in a second data group corresponding to the number of GPUs, including: Get the total number of data groups in the first data group of the merged data; The number of data groups allocated to the first data group in each group is determined based on the number of graphics cards and the number of data groups in the server used for model training. Select the first data group corresponding to the allocated quantity from the merged data to obtain the second data group corresponding to the number of graphics cards. The second data group includes the first data group of the allocated quantity.

[0065] The allocation quantity for each graphics card can be obtained by comparing the total number of the first data group in the merged data with the number of graphics cards. The allocation quantity is also the number of the first data group in each second data group. The first data group with the corresponding allocation quantity is randomly selected from the merged data to obtain the second data group, which helps to improve the efficiency of data allocation.

[0066] Optionally, when using the target data for model training, distributed data parallel training can be adopted, where each row of the target data is allocated to a corresponding graphics card to perform model training.

[0067] Currently, training models generally employ Distributed Data Parallel (DDP) data scheduling. Therefore, data scheduling involves batch and world_size. For example, if the server uses GPUs, then batch is the amount of data in a training batch, and world_size is the number of GPUs on the GPU server. DDP will allocate different data to each GPU, so it is necessary to group the data into batches and world_sizes to ensure that there is only one modality of data in each batch, and that the data are of similar length.

[0068] In DDP mode, the data used in each training step is actually world_size * batch, and the data is called in parallel according to world_size. That is, the code will first take the data of world_size size and execute batch times.

[0069] For example, the merged data is [[D3,D4],[D7,D8],[D1,D2],[D5,D6]]. All first data groups in the merged data are divided into world_size groups, where world_size=2. For instance, [D3,D4] and [D1,D2] are divided into world_size_1 = ([D3,D4,D1,D2]), and [D7,D8] and [D5,D6] are divided into world_size_2 = ([D7,D8,D5,D6]). Then, world_size_1 is sent to the first graphics card, and world_size_2... Sending ize_2 to the second graphics card will not disrupt the expected ordered arrangement. world_size is unfolded and flattened into [D3,D7,D4,D8,D1,D5,D2,D6]. Furthermore, the batches of world_size will be shuffled to further ensure randomness. For example, after shuffling, world_size_1 = ([D7,D8,D1,D2]) and world_size_2 = ([D3,D4,D5,D6]), and after flattening, it becomes [D7,D3,D8,D4,D1,D5,D2,D6].

[0070] The entire process of this application is as follows: 1) First, we will group the data according to its modality information to ensure that each batch contains only one modality, thereby enabling efficient parallel operations related to modality. 2) After grouping, the data for each modality will be sorted by length. 3) After sorting, the data will be split into batches to ensure that the data lengths within a batch are similar. Then, the data will be shuffled in batches to ensure the randomness of the data. 4) Next, the data from various modalities will be merged. Multimodal training is generally based on DDP data scheduling. The data is not extracted directly in batch order. It also needs to be grouped into world size groups and shuffled to obtain the final target data.

[0071] In this embodiment, the core of multimodal data batch / world_size grouping and shuffle is to group the data by modality and sort it by length. Modality grouping is to distinguish different modalities and facilitate parallel processing of modalities. Length sorting is to make the data lengths within a batch as close as possible, thereby reducing the computational loss caused by padding. However, data cannot be correctly scheduled simply by grouping and sorting. It needs to be combined into a specific order to achieve the target effect. Moreover, the data also needs to have a certain degree of randomness to avoid overfitting during training. Therefore, it is necessary to combine and shuffle the data according to the data scheduling method.

[0072] Corresponding to the multimodal data combination method for model training in the above embodiment, Figure 5 This paper shows a structural block diagram of a multimodal data combination device for model training provided in Embodiment 5 of this application. The aforementioned multimodal data combination device for model training can be applied to… Figure 1 The server-side component connects to the client, database, etc., via a corresponding computer device to obtain and process the data. For ease of explanation, only the parts relevant to the embodiments of this application are shown.

[0073] See Figure 5 The multimodal data combination device for model training includes: Modality grouping module 51 is used to acquire training data of N modes, group all training data according to mode, and obtain the modality grouping result for each mode, where N is an integer greater than 1; The data sorting module 52 is used to sort all the training data in any modality grouping result according to the data length in ascending or descending order, so as to obtain the sequence data of the corresponding modality grouping result. The batch segmentation module 53 is used to segment the sequence data using a preset batch data amount to obtain at least one first data group of the corresponding modality grouping result. Each first data group includes training data corresponding to the preset batch data amount. The data combination module 54 is used to traverse all modality grouping results, obtain the first data group corresponding to each modality grouping result, combine all the first data groups to obtain merged data, and use the merged data to train the model.

[0074] Optionally, the multimodal data combination device for model training also includes: The first rearrangement module is used to divide the sequence data using a preset batch of data to obtain at least one first data group of the corresponding modal grouping result, and then randomly arrange all the first data groups of the corresponding modal grouping result to obtain the rearranged first data group of the corresponding modal grouping result. The data combination module 54 includes: The data combination unit is used to traverse all modal grouping results, obtain the first data group after rearrangement for each modal grouping result, and combine all rearranged first data groups to obtain the merged data.

[0075] Optionally, the batch splitting module 53 includes: The segmentation unit is used to segment the sequence data sequentially from front to back according to a preset batch data volume, to obtain M first data groups, where M is an integer greater than zero.

[0076] Optionally, the multimodal data combination device for model training also includes: The graphics card grouping module is used to group the merged data after merging all the first data groups to obtain the merged data, and then group the merged data according to the number of graphics cards in the model training server to obtain the second data group corresponding to the number of graphics cards. The target data combination module is used to flatten all the second data groups to obtain the flattened data as the target data. The number of rows of the target data is equal to the number of graphics cards, and the number of columns of the target data is equal to the preset batch size of data. The target data is used to train the model.

[0077] Optionally, the multimodal data combination device for model training also includes: The second rearrangement module is used to group the merged data according to the number of GPUs in the server used for model training, and then to perform random permutation among all the second data groups, taking the first data group as the unit, to obtain the rearranged second data groups. The target data combination module includes: The target data combination unit is used to tile all rearranged second data groups to obtain the tiled data as the target data.

[0078] Optionally, the graphics card grouping module includes: The quantity acquisition unit is used to acquire the total quantity of the first data group in the merged data; The quantity allocation unit is used to determine the allocation quantity of the first data group in each group based on the number of graphics cards and the number of data groups in the model training server. The graphics card grouping unit is used to select a first data group corresponding to the allocated quantity from the merged data to obtain a second data group corresponding to the number of graphics cards. The second data group includes the first data group with the allocated quantity.

[0079] Optionally, when using the target data for model training, distributed data parallel training can be adopted, where each row of the target data is allocated to a corresponding graphics card to perform model training.

[0080] It should be noted that the information interaction and execution process between the above modules, units, and sub-units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0081] Figure 6 This is a schematic diagram of the structure of a computer device provided in Embodiment Six of this application. Figure 6 As shown, the computer device of this embodiment includes: at least one processor ( Figure 6 Only one is shown in the diagram), a memory, and a computer program stored in the memory and executable on at least one processor. When the processor executes the computer program, it implements the steps of any of the above-described multimodal data combination methods for model training or embodiments of the multimodal data combination method for model training.

[0082] This computer device may include, but is not limited to, a processor and memory. Those skilled in the art will understand that... Figure 6 The examples of computer devices are merely examples and do not constitute a limitation on computer devices. Computer devices may include more or fewer components than shown in the illustration, or combinations of certain components, or different components, such as network interfaces, displays, and input devices.

[0083] The processor referred to can be a CPU, but it can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0084] Memory includes readable storage media, internal memory, etc., wherein internal memory can be the RAM of a computer device, providing an environment for the operation of the operating system and computer-readable instructions stored in the readable storage media. The readable storage media can be the hard drive of a computer device, or in other embodiments, it can be an external storage device of the computer device, such as a plug-in hard drive, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, memory can include both internal storage units and external storage devices of the computer device. Memory is used to store the operating system, applications, bootloader, data, and other programs, such as program code for computer programs. Memory can also be used to temporarily store data that has been output or will be output.

[0085] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above device can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium can include at least: any entity or device capable of carrying computer program code, a recording medium, a computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0086] The implementation of all or part of the processes in the methods of the above embodiments can also be accomplished by a computer program product. When the computer program product is run on a computer device, it enables the computer device to execute the steps in the above method embodiments.

[0087] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0088] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0089] In the embodiments provided in this application, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0090] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0091] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for combining multimodal data for model training, characterized in that, include: Obtain training data for N modalities, group all training data according to modality, and obtain the modality grouping results for each modality, where N is an integer greater than 1; For any modality grouping result, sort all the training data in the modality grouping result in ascending or descending order of data length to obtain the sequence data corresponding to the modality grouping result; Using a preset batch of data, the sequence data is segmented to obtain at least one first data group corresponding to the modality grouping result, and each first data group includes training data corresponding to the preset batch of data. Iterate through all modality grouping results to obtain the first data group corresponding to each modality grouping result. Merge all the first data groups to obtain the merged data, which is used to train the model.

2. The method for combining multimodal data for model training according to claim 1, characterized in that, After segmenting the sequence data using a preset batch of data to obtain at least one first data group corresponding to the modality grouping result, the method further includes: Randomly arrange all the first data groups corresponding to the modal grouping results to obtain the rearranged first data groups corresponding to the modal grouping results; The process involves iterating through all modality grouping results to obtain the first data group corresponding to each modality grouping result. All first data groups are then merged to obtain the merged data, including: Iterate through all modal grouping results to obtain the first data group after rearrangement for each modal grouping result. Merge all rearranged first data groups to obtain the merged data.

3. The method for combining multimodal data for model training according to claim 1 or 2, characterized in that, The step of segmenting the sequence data using a preset batch of data to obtain at least one first data group corresponding to the modality grouping result includes: Based on a preset batch of data, the sequence data is divided sequentially from front to back to obtain M first data groups, where M is an integer greater than zero.

4. The method for combining multimodal data for model training according to claim 1, characterized in that, After combining all the first data to obtain the merged data, the process also includes: The merged data is grouped according to the number of GPUs in the server used for model training, resulting in a second data group corresponding to the number of GPUs. All second data groups are tiled to obtain the target data. The number of rows in the target data is the number of graphics cards, and the number of columns in the target data is the preset batch size. The target data is used to train the model.

5. The method for combining multimodal data for model training according to claim 4, characterized in that, After grouping the merged data according to the number of graphics cards in the server used for model training to obtain a second data group corresponding to the number of graphics cards, the method further includes: Randomly arrange the data among all the second data groups, taking the first data group as a unit, to obtain rearranged second data groups; The step of tiling all the second data groups to obtain the tiled data as the target data includes: Tile all rearranged second data groups to obtain the tiled data as the target data.

6. The method for combining multimodal data for model training according to claim 4, characterized in that, The merged data is grouped according to the number of graphics cards in the server used for model training, resulting in a second data group corresponding to the number of graphics cards, including: Obtain the total number of the first data group in the merged data; The number of data groups allocated to the first data group in each group is determined based on the number of graphics cards in the server used for model training and the number of data groups. A first data group corresponding to the allocated quantity is selected from the merged data to obtain a second data group corresponding to the number of graphics cards, wherein the second data group includes the first data group of the allocated quantity.

7. The method for combining multimodal data for model training according to claim 4, characterized in that, When using the target data for model training, distributed data parallel training is adopted, in which a row of data from the target data is allocated to the corresponding graphics card to perform model training.

8. A multimodal data combination device for model training, characterized in that, include: The modality grouping module is used to acquire training data for N modalities, group all training data according to modality, and obtain the modality grouping result for each modality, where N is an integer greater than 1; The data sorting module is used to sort all training data in any modality grouping result in ascending or descending order of data length to obtain sequence data corresponding to the modality grouping result. The batch segmentation module is used to segment the sequence data using a preset batch data volume to obtain at least one first data group corresponding to the modality grouping result, wherein each first data group includes training data corresponding to the preset batch data volume. The data combination module is used to traverse all modality grouping results, obtain the first data group corresponding to each modality grouping result, and combine all the first data groups to obtain the merged data, which is used to train the model.

9. A computer device, characterized in that, The computer device includes a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the multimodal data combination method for model training as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the multimodal data combination method for model training as described in any one of claims 1 to 7.

Images