An input data processing method and device of a multi-modal model

Abstract

The application provides an input data processing method and device of a multi-modal model. The method comprises the following steps: a first data parallel domain obtains a plurality of input vectors corresponding to the first data parallel domain, the first data parallel domain being any data parallel domain in M data parallel domains, and the M data parallel domains achieving the redistribution of the input vectors in the M data parallel domains through parallel communication between the M data parallel domains; the total length difference of the input vectors in any two data parallel domains after the redistribution is less than a first threshold; and the first data parallel domain calculates the input vectors after the redistribution in the first data parallel domain by using a multi-modal model deployed on the first data parallel domain. Since the total length difference of the input vectors after the redistribution in each data parallel domain is small, the load of each data parallel domain can reach a balanced state when each data parallel domain calculates the input vectors after the redistribution in each data parallel domain.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for processing input data of a multimodal model. Background Technology

[0002] With the rapid development of artificial intelligence (AI) technology, multimodal models are widely used. When training multimodal models using batch training data, to accelerate the training process, the training data in a batch can be distributed across multiple computing devices, which then train the multimodal model based on their allocated data. However, since the data lengths of the individual training data points within a batch are not identical, the total length of the training data allocated to different computing devices will vary, leading to an uneven workload on each device during multimodal model training.

[0003] Solving the problem of uneven load distribution among various computing devices during multimodal model training is a pressing technical issue that needs to be addressed. Summary of the Invention

[0004] This application provides a method and apparatus for processing input data of a multimodal model, which is used to solve the problem of unbalanced load on various computing devices when training a multimodal model.

[0005] Firstly, this application provides a method for processing input data of a multimodal model. The execution body of this method is M data parallel domains in an AI cluster 102. The method includes: a first data parallel domain acquiring multiple input vectors corresponding to the first data parallel domain, wherein the first data parallel domain is any one of the M data parallel domains, and the M data parallel domains correspond to N input vectors; M and N are both greater than 1; then, the M data parallel domains redistribute the input vectors in the M data parallel domains through parallel communication between the M data parallel domains; the total length difference between any two input vectors in the redistributed data parallel domains is less than a first threshold; finally, the first data parallel domain uses the multimodal model deployed on the first data parallel domain to calculate the redistributed input vectors in the first data parallel domain, and each of the M data parallel domains deploys a complete and identical multimodal model.

[0006] In the above method, each of the M data parallel domains does not need to wait for the redistribution of input vectors in other data parallel domains to complete. The M data parallel domains directly achieve the redistribution of input vectors in the M data parallel domains through parallel communication, which is beneficial to improving redistribution efficiency. Furthermore, since the total length difference of the redistributed input vectors in each data parallel domain is small, the load of each data parallel domain can be balanced when calculating the redistributed input vectors in each data parallel domain.

[0007] In one possible design, the M parallel data domains redistribute the input vectors within them through parallel communication. This includes: when the total length difference between the input vectors in any two of the M parallel data domains is greater than or equal to a first threshold, the M parallel data domains redistribute the input vectors through parallel communication. Thus, when the total length difference between the input vectors in any two of the M parallel data domains is small, input vector redistribution is avoided, thereby accelerating subsequent multimodal model training or inference tasks.

[0008] In one possible design, before the redistribution of input vectors in the M data parallel domains is achieved through parallel communication between the M data parallel domains, the process includes: a management resource determining the target data parallel domain corresponding to each input vector in the M data parallel domains based on the length of the input vectors in the M data parallel domains; and the redistribution of input vectors in the M data parallel domains through parallel communication between the M data parallel domains, which includes: the M data parallel domains redistributing each input vector in the M data parallel domains based on the target data parallel domain corresponding to each input vector in the M data parallel domains through parallel communication between the M data parallel domains. In this design, the management resource first determines the target data parallel domain corresponding to each input vector in the M data parallel domains, and then the M data parallel domains send each input vector to the target data parallel domain corresponding to each input vector, thus accurately achieving the redistribution of each input vector. In this process, each of the M parallel data domains does not need to wait for the redistribution of input vectors in other parallel data domains to be completed. Instead, it can quickly redistribute the input vectors in the M parallel data domains through parallel communication between them, which helps to improve redistribution efficiency.

[0009] In one possible design, M data parallel domains, based on the target data parallel domain corresponding to each input vector in the M data parallel domains, achieve the redistribution of each input vector in the M data parallel domains through parallel communication between the M data parallel domains. This includes: the M data parallel domains sending each input vector in the M data parallel domains to the target data parallel domain corresponding to each input vector in parallel through parallel communication between the M data parallel domains, thereby achieving the redistribution of each input vector in the M data parallel domains. This design provides a specific method for input vector redistribution in M ​​data parallel domains.

[0010] In one possible design, the management resource determines the target data parallel domain corresponding to each input vector in the M data parallel domains based on the lengths of the input vectors in the M data parallel domains. This includes: the management resource determines the ascending order index of each input vector in each of the S1 data parallel domains based on the lengths of the input vectors in the S1 data parallel domains; when M is even, S1 is (M / 2); when M is odd, S1 is... Based on the lengths of the input vectors in the remaining S2 data parallel domains, determine the descending order of the input vectors in each of the remaining S2 data parallel domains; when M is even, S2 is (M / 2); when M is odd, S2 is... Based on the ascending order of the input vectors in the S1 data parallel domains and the descending order of the input vectors in the remaining S2 data parallel domains, the target data parallel domain corresponding to each input vector in the M data parallel domains is determined. This design provides a method for determining the target data parallel domain corresponding to the input vectors in the data parallel domains, and the total length difference of the input vectors in the target data parallel domains determined by this design is small.

[0011] In one possible design, the management resource determines the target data parallel domain corresponding to each input vector in the M data parallel domains based on the length of the input vectors in the M data parallel domains. This includes: for one data parallel domain among the M data parallel domains, the management resource performs the following steps: the management resource determines the sorting index of each input vector in the data parallel domain based on the length of each input vector in the data parallel domain; the management resource determines the target data parallel domain corresponding to each input vector in the data parallel domain based on the sorting index of each input vector in the data parallel domain. This design provides a method for determining the target data parallel domain corresponding to input vectors in a data parallel domain, and the total length difference of the input vectors in the various target data parallel domains determined by this design is relatively small.

[0012] In one possible design, the management resource determines the target data parallel domain corresponding to each input vector in the M data parallel domains based on the lengths of the input vectors in the M data parallel domains. This includes: the management resource determining M input vector groups from the input vectors in the M data parallel domains; each input vector group includes (N / M) encoded vectors, and the difference between the total length of the (N / M) input vectors in each input vector group and the average length corresponding to the data parallel domain is less than a second threshold; the average length corresponding to the data parallel domain is determined based on the N input vectors; the management resource determines the target data parallel domain corresponding to each of the M input vector groups based on the M data parallel domains, wherein the target data parallel domain corresponding to the input vectors in each input vector group is the same. This design provides a method for determining the target data parallel domain corresponding to the input vectors in the data parallel domains, and the total length of the input vectors in the target data parallel domains determined by this design has small differences.

[0013] In one possible design, the management resource determines the target data parallel domain corresponding to each input vector in the M data parallel domains based on the length of each input vector in the M data parallel domains. This includes: the management resource sorting the input vectors in the M data parallel domains as a whole according to their lengths to obtain the sorting index of each input vector in the M data parallel domains; the management resource repeats the following steps until each input vector in the M data parallel domains corresponds to a target data parallel domain: the management resource selects the input vectors with the smallest sorting index (N / M) from the input vectors in the M data parallel domains, and selects one data parallel domain from the M data parallel domains as the target data parallel domain corresponding to the input vectors with the smallest sorting index (N / M). This design provides a method for determining the target data parallel domain corresponding to input vectors in data parallel domains, allowing input vectors of similar lengths to correspond to the same target data parallel domain. In inference scenarios, this design allows multimodal models to output inference results corresponding to shorter input vectors earlier, which is beneficial for improving the user experience in inference scenarios.

[0014] Secondly, this application also provides an input data processing apparatus for a multimodal model, which includes methods for performing the methods provided in the first aspect above. The input data processing apparatus for the multimodal model includes: an acquisition module, a redistribution module, a calculation module, and a determination module.

[0015] The acquisition module is used to acquire multiple input vectors corresponding to the first data parallel domain among M data parallel domains; the first data parallel domain is any one of the M data parallel domains, and the M data parallel domains correspond to N input vectors; M and N are both greater than 1.

[0016] The redistribution module is used to redistribute the input vectors in the M data parallel domains through parallel communication between the M data parallel domains; the difference in the total length of the input vectors in any two data parallel domains after redistribution is less than a first threshold.

[0017] The computation module is used to compute the redistributed input vector in the first data parallel domain using a multimodal model deployed on the first data parallel domain. Each of the M data parallel domains deploys a complete and identical multimodal model.

[0018] In one possible design, the redistribution module is specifically used to: redistribute the input vectors in the M data parallel domains through parallel communication between the M data parallel domains when the total length difference between any two data parallel domains is greater than or equal to a first threshold.

[0019] In one possible design, a determination module is used to determine the target data parallel domain corresponding to each input vector in the M data parallel domains based on the length of the input vectors in the M data parallel domains.

[0020] The redistribution module is specifically used to: redistribute the input vectors in the M data parallel domains according to the target data parallel domain corresponding to each input vector in the M data parallel domains, through parallel communication between the M data parallel domains.

[0021] In one possible design, the redistribution module is specifically used to: send each input vector in the M data parallel domains in parallel to the target data parallel domain corresponding to each input vector through parallel communication between the M data parallel domains, thereby realizing the redistribution of each input vector in the M data parallel domains.

[0022] In one possible design, the determining module is specifically used to: determine the ascending order index of each input vector in each of the S1 data parallel domains based on the length of each input vector in each of the S1 data parallel domains; when M is even, S1 is (M / 2); when M is odd, S1 is... Based on the lengths of the input vectors in the remaining S2 data parallel domains, determine the descending order of the input vectors in each of the remaining S2 data parallel domains; when M is even, S2 is (M / 2); when M is odd, S2 is... Based on the ascending order of the input vectors in the S1 data parallel domains and the descending order of the input vectors in the remaining S2 data parallel domains, determine the target data parallel domain corresponding to each input vector in the M data parallel domains.

[0023] In one possible design, the module is specifically used to: for one of the M data parallel domains, perform the following steps: determine the sorting index of each input vector in the data parallel domain based on the length of each input vector in the data parallel domain; determine the target data parallel domain corresponding to each input vector in the data parallel domain based on the sorting index of each input vector in the data parallel domain.

[0024] In one possible design, the determining module is specifically used to: determine M input vector groups from input vectors in M ​​data parallel domains; each input vector group includes (N / M) encoded vectors, and the difference between the total length of the (N / M) input vectors in each input vector group and the average length corresponding to the data parallel domain is less than a second threshold; the average length corresponding to the data parallel domain is determined based on the N input vectors; and based on the M data parallel domains, determine the target data parallel domains corresponding to the M input vector groups respectively, wherein the target data parallel domains corresponding to the input vectors in each input vector group are the same.

[0025] In one possible design, a module is defined as follows: based on the length of each input vector in the M data parallel domains, the input vectors in the M data parallel domains are sorted as a whole to obtain the sorting index of each input vector in the M data parallel domains; the following steps are repeated until each input vector in the M data parallel domains corresponds to a target data parallel domain: selecting the input vectors with the smallest sorting index (N / M) from the input vectors in the M data parallel domains, and selecting a data parallel domain from the M data parallel domains as the target data parallel domain corresponding to the input vectors with the smallest sorting index (N / M).

[0026] Thirdly, this application also provides a computer program product containing instructions that, when executed by a computing system, cause the computing system to perform the method provided in the first aspect above.

[0027] Fourthly, this application also provides a computer-readable storage medium including computer program instructions, which, when executed by a computing system, enable the computing system to perform the method provided in the first aspect above.

[0028] Fifthly, this application also provides a computing device, including a processor and a memory, wherein the memory stores computer-executable instructions; the processor is used to execute the computer-executable instructions stored in the memory, and the processor performs the method provided in the first aspect above.

[0029] Sixthly, this application also provides a chip containing instructions that, when executed by a computing system, cause the computing system to perform the method provided in the first aspect above.

[0030] In a seventh aspect, this application provides a chip system containing instructions that, when executed by a computing system, cause the computing system to perform the method provided in the first aspect above. This chip system may be composed of chips or may include chips and other discrete devices. Attached Figure Description

[0031] Figure 1A A system architecture diagram provided for an embodiment of this application;

[0032] Figure 1B An architecture diagram of an AI cluster is provided for an embodiment of this application;

[0033] Figure 2 A flowchart illustrating an input data processing method for a multimodal model provided in an embodiment of this application;

[0034] Figure 3 This application provides a schematic diagram of the structure of an input vector in a data parallel domain.

[0035] Figure 4 A flowchart illustrating a redistribution method provided in an embodiment of this application;

[0036] Figure 5 A flowchart illustrating a method for determining a target data parallel domain, provided in an embodiment of this application;

[0037] Figure 6 A flowchart illustrating a method for determining a target data parallel domain, provided in an embodiment of this application;

[0038] Figure 7 A flowchart illustrating a method for determining a target data parallel domain, provided in an embodiment of this application;

[0039] Figure 8 A flowchart illustrating a method for determining a target data parallel domain, provided in an embodiment of this application;

[0040] Figure 9 A flowchart illustrating a method for determining a target data parallel domain, provided in an embodiment of this application;

[0041] Figure 10 A flowchart illustrating a method for determining a target data parallel domain, provided in an embodiment of this application;

[0042] Figure 11 This application provides a schematic diagram of the structure of a redistributed input vector.

[0043] Figure 12 This is a schematic diagram of the structure of an input data processing device for a multimodal model provided in an embodiment of this application. Detailed Implementation

[0044] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. The terms "first," "second," and corresponding reference numerals in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0045] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0046] A multimodal model is a model that can understand and process multiple types of data. These multiple types of data include, but are not limited to, text, images, audio, and video.

[0047] Data parallelism is a widely used technique in distributed systems that aims to improve computational efficiency and speed by dividing a dataset into multiple subsets and processing these subsets in parallel on different computing devices. In deep learning and large-scale machine learning training, data parallelism can significantly accelerate the training speed of artificial intelligence models and improve model performance.

[0048] The present application will now be described in further detail with reference to the accompanying drawings. The specific operating methods and functional descriptions in the method embodiments can also be applied to the device embodiments or system embodiments.

[0049] Figure 1A This paper illustrates a possible system architecture diagram for the input data processing method of the multimodal model provided in the embodiments of this application. Figure 1A The system architecture shown includes a terminal device 101 and an AI cluster 102, which can be connected via wired or wireless means.

[0050] The components of this system architecture are described below.

[0051] (1) Terminal equipment 101

[0052] The user can input the data parallelism M and the batch size S in the terminal device 101, and the terminal device 101 will send the data parallelism M and the batch size S to the AI ​​cluster 102; correspondingly, the AI ​​cluster 102 will receive the data parallelism M and the batch size S from the terminal device 101. Both S and M are greater than 1.

[0053] Among them, the data parallelism is used to determine the number of data parallel domains in AI cluster 102. When the data parallelism is M, AI cluster 102 is divided into M data parallel domains, and each data parallel domain is deployed with a complete and identical multimodal model.

[0054] A batch refers to a set of input data that is input to M data-parallel domains at once. For example, during the training of a multimodal model, the dataset contains a large amount of input data. The input data in the dataset can be divided into multiple batches, and N input data from one batch are input to the M data-parallel domains at a time. Each data-parallel domain multimodal model processes (N / M) input data from one batch.

[0055] In this application embodiment, terminal device 101 is a device with wireless transceiver capabilities, which can be deployed on land, including indoors or outdoors, handheld, wearable, or vehicle-mounted; it can also be deployed on water (such as ships); and it can also be deployed in the air (e.g., on airplanes, balloons, and satellites). Terminal devices can be mobile phones, tablets, computers, virtual reality (VR) terminals, augmented reality (AR) terminals, vehicle-mounted terminals, wireless terminals in self-driving vehicles, wireless terminals in remote medical care, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, wearable terminals, etc. The embodiments of this application do not limit the application scenarios.

[0056] Terminal device 101 can be fixed or mobile. It is understood that all or part of the functions of terminal device 101 in this application can also be implemented by software functions running on hardware, or by virtualization functions instantiated on a platform (e.g., a cloud platform).

[0057] (2) AI Cluster 102

[0058] The architecture of AI cluster 102 can be referenced. Figure 1B The AI ​​cluster 102 includes management resources, computing resources, a storage medium resource pool, external storage, and memory. The management resources and computing resources can communicate with each other, both can access memory, the management resources can access external storage, and the computing resources can access the storage medium resource pool.

[0059] The management resources include at least one central processing unit (CPU).

[0060] Computing resources include multiple acceleration devices. These acceleration devices can be graphics processing units (GPUs), tensor processing units (TPUs), neural network processing units (NPUs), deep learning processing units (DPUs), etc., and are not limited here.

[0061] Computing resources can access a storage medium resource pool, which includes multiple storage devices. For example, the storage devices can be high bandwidth memory (HBM), etc., without limitation.

[0062] External storage can be used to store datasets, where the input data is used for training or inference of multimodal models. Any input data in the dataset can include multiple data sources such as text, images, audio, and video. External storage is a non-volatile storage medium, which can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory, etc.

[0063] Memory can be used to temporarily store input data sent from secondary storage. Memory is a volatile storage medium. Volatile storage media can be random access memory (RAM). By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), flash EPROM (FE), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced synchronous DRAM (ESDRAM), synchronous linked DRAM (SLDRAM), and direct rambus RAM (DR RAM), as well as newer memories such as resistive random access memory (ReRAM), ferroelectric random access memory (FeRAM), phase change random access memory (PCRAM), or magnetic random access memory (MRAM).

[0064] After obtaining the data parallelism M, the CPU in the management resources of AI cluster 102 can divide the multiple acceleration devices in the computing resources into M data parallel domains, where each data parallel domain includes at least one acceleration device.

[0065] It should be understood that a CPU managing resources can manage one data parallel domain or multiple data parallel domains, without limitation. A data parallel domain can be managed by one CPU or multiple CPUs, without limitation. For ease of explanation later, this application will use the example of one CPU managing one data parallel domain for illustrative purposes.

[0066] After determining the number of input data and the batch size S in the dataset stored in the external storage, the CPU in the AI ​​cluster 102 can calculate and determine the number of input data N in a batch based on the number of input data and the batch size S in the dataset. After determining the number of input data N in a batch, the N input data in a batch can be processed through any of the following implementation methods A1, A2, and A3.

[0067] In implementation method A1, when there is a master CPU in the management resources of the AI ​​cluster 102, the master CPU can be any CPU in the management resources. The master CPU sends a data dispatch instruction to the secondary storage, wherein the data dispatch instruction is used to instruct the secondary storage to send N input data to the main memory. After receiving the data dispatch instruction, the secondary storage sends N input data to the main memory. After the secondary storage sends N input data to the main memory, the secondary storage sends a first response instruction to the master CPU, wherein the first response instruction is used to indicate that the secondary storage has successfully sent N input data to the main memory. Here, N is greater than 1.

[0068] After receiving the first response instruction, the main CPU sends a first fetch instruction to any one of the M data parallel domains. This first fetch instruction instructs the data parallel domain to fetch N input data points from memory. After receiving the first fetch instruction, the data parallel domain fetches the N input data points from memory and then stores them as a batch in the memory of the storage medium resource pool. After fetching the N input data points, the data parallel domain sends a second response instruction to the main CPU, indicating that it has successfully fetched the N input data points.

[0069] After receiving the second response instruction, the main CPU sends a vectorization processing instruction to the data parallel domain. This instruction instructs the data parallel domain to perform vectorization processing on the input data. Upon receiving the vectorization processor instruction, the data parallel domain performs vectorization processing on the N input data, resulting in N input vectors. These N input vectors can also be stored in the memory within the storage medium resource pool.

[0070] Subsequently, the data parallel domain sends (N / M) input vectors to the other (M-1) data parallel domains respectively. After each of the M data parallel domains has obtained (N / M) input vectors, the M data parallel domains redistribute the input vectors in the M data parallel domains through parallel communication between the M data parallel domains. Finally, the first data parallel domain in the M data parallel domains uses the multimodal model deployed on the first data parallel domain to calculate the redistributed input vectors in the first data parallel domain. The first data parallel domain can be any one of the M data parallel domains.

[0071] In implementation method A2, if there is a master CPU in the management resources of AI cluster 102, the master CPU can be any CPU in the management resources. The master CPU sends a data dispatch instruction to the secondary storage, wherein the data dispatch instruction is used to instruct the secondary storage to send N input data to the main memory. After receiving the data dispatch instruction, the secondary storage sends N input data to the main memory. After the secondary storage sends N input data to the main memory, the secondary storage sends a first response instruction to the master CPU, wherein the first response instruction is used to indicate that the secondary storage has successfully sent N input data to the main memory.

[0072] After receiving the first response instruction, the main CPU sends a second fetch instruction to each of the M data parallel domains. The second fetch instruction instructs each data parallel domain to fetch (N / M) input data from memory. Taking any one of the M data parallel domains as an example, after receiving the second fetch instruction, this data parallel domain fetches (N / M) input data from memory and then stores the fetched (N / M) input data in the memory of the storage medium resource pool. After fetching (N / M) input data, this data parallel domain sends a third response instruction to the main CPU, indicating that it has successfully fetched (N / M) input data.

[0073] After receiving the third response instruction, the main CPU sends a vectorization processing instruction to the data parallel domain. This instruction instructs the data parallel domain to perform vectorization processing on the input data. Upon receiving the vectorization processor instruction, the data parallel domain performs vectorization processing on (N / M) input data, resulting in (N / M) input vectors. These (N / M) input vectors can also be stored in the memory within the storage medium resource pool.

[0074] After each of the M data parallel domains obtains (N / M) input vectors, the M data parallel domains redistribute the input vectors through parallel communication between the M data parallel domains. Finally, the first data parallel domain among the M data parallel domains uses a multimodal model deployed on the first data parallel domain to calculate the redistributed input vectors in the first data parallel domain. The first data parallel domain can be any one of the M data parallel domains.

[0075] In implementation method A3, where multiple main CPUs manage resources in AI cluster 102, and each main CPU manages one or more data parallel domains, any one of the main CPUs sends a data delivery instruction to the secondary storage, instructing the secondary storage to send N input data to the main memory. Upon receiving the data delivery instruction, the secondary storage sends the N input data to the main memory. After sending the N input data, the secondary storage sends a first response instruction to the main CPU, indicating that it has successfully sent the N input data to the main memory. Afterward, the main CPU can send the first response instruction to other main CPUs.

[0076] After multiple master CPUs receive the first response instruction, taking one of the master CPUs as an example, this master CPU issues a second fetch instruction to each of its managed data parallel domains. The second fetch instruction instructs the data parallel domain to fetch (N / M) input data from memory. Taking any data parallel domain managed by this master CPU as an example, after receiving the second fetch instruction, this data parallel domain fetches (N / M) input data from memory and then stores the fetched (N / M) input data in the memory of the storage medium resource pool. After fetching (N / M) input data, the data parallel domain sends a third response instruction to the master CPU, indicating that the data parallel domain has successfully fetched (N / M) input data.

[0077] After receiving the third response instruction, the main CPU sends a vectorization processing instruction to the data parallel domain. This instruction instructs the data parallel domain to perform vectorization processing on the input data. Upon receiving the vectorization processor instruction, the data parallel domain performs vectorization processing on (N / M) input data, resulting in (N / M) input vectors. These (N / M) input vectors can also be stored in the memory within the storage medium resource pool.

[0078] After each of the M data parallel domains obtains (N / M) input vectors, the M data parallel domains redistribute the input vectors through parallel communication between the M data parallel domains. Finally, the first data parallel domain among the M data parallel domains uses a multimodal model deployed on the first data parallel domain to calculate the redistributed input vectors in the first data parallel domain. The first data parallel domain can be any one of the M data parallel domains.

[0079] It should be understood that, based on Figure 1A Chinese system architecture and Figure 1B Modifications and variations to the architecture diagram of the AI ​​cluster do not depart from the scope of protection of this application.

[0080] Combination Figure 1A The system architecture diagram shown and Figure 1B The diagram shows the architecture of an AI cluster. This application embodiment provides a method for processing input data of a multimodal model. This method can be... Figure 1B The AI ​​cluster 102 shown contains M data parallel domains or management resources for execution. (See also...) Figure 2 As shown, the process of this method includes:

[0081] S201, the first data parallel domain acquires multiple input vectors corresponding to the first data parallel domain, wherein the first data parallel domain is any one of the M data parallel domains.

[0082] In S201 above, each of the M data parallel domains acquires the same number of input vectors. With each data parallel domain corresponding to (N / M) input vectors, the M data parallel domains correspond to N input vectors.

[0083] In this embodiment, the first data parallel domain can obtain multiple input vectors by referring to any one of the above embodiments A1, A2 or A3.

[0084] The explanation will take the acquisition of multiple input vectors in each data parallel domain according to implementation method A1 as an example, referring to... Figure 3 The data parallelism is set to 2, therefore AI cluster 102 includes two data parallel domains: data parallel domain 1 and data parallel domain 2. In the management resources, the main CPU sends a data dispatch instruction to the secondary storage, which instructs the secondary storage to send 8 input data points to the main memory. After receiving the data dispatch instruction, the secondary storage sends 8 input data points to the main memory. After sending the 8 input data points to the main memory, the secondary storage sends a first response instruction to the main CPU, which indicates that the secondary storage has successfully sent the 8 input data points to the main memory.

[0085] After receiving the first response instruction, the main CPU sends a first fetch instruction to data parallel domain 1, which instructs the data parallel domain to fetch eight input data items from memory. After receiving the first fetch instruction, data parallel domain 1 fetches the eight input data items from memory and then stores them as a batch in the memory of the storage medium resource pool. After data parallel domain 1 has fetched the eight input data items, it sends a second response instruction to the main CPU, indicating that data parallel domain 1 has successfully fetched the eight input data items.

[0086] After receiving the second response instruction, the main CPU sends a vectorization processing instruction to data parallel domain 1. This instruction instructs the data parallel domain to perform vectorization processing on the input data. After receiving the vectorization processor instruction, data parallel domain 1 performs vectorization processing on the eight input data points, resulting in eight input vectors. These eight input vectors can also be stored in the memory within the storage medium resource pool. Each input vector from 1 to 8 is a two-dimensional matrix of size w*h, where w and h are both greater than 0. The h values ​​of input vectors 1 to 8 are all the same. The w value of input vector 1 is the same as that of input vector 2, the w value of input vector 3 is the same as that of input vector 4, the w value of input vector 5 is the same as that of input vector 6, and the w value of input vector 7 is the same as that of input vector 8. The w value of input vector 1 is greater than that of input vector 3, the w value of input vector 3 is greater than that of input vector 5, and the w value of input vector 5 is greater than that of input vector 7.

[0087] Subsequently, data parallel domain 1 retains input vectors 1 to 4 and sends input vectors 5 to 8 to data parallel domain 2. Based on Figure 3 As can be seen from the values ​​of w for each input vector, the total length of the input vectors in data parallel domain 1 is much greater than the total length of the input vectors in data parallel domain 2. Specifically, the total length of the input vectors in data parallel domain 1 is the sum of the lengths of input vectors 1 to 4, and the total length of the input vectors in data parallel domain 2 is the sum of the lengths of input vectors 5 to 8.

[0088] S202, M parallel data domains achieve the redistribution of input vectors in the M parallel data domains through parallel communication between the M parallel data domains, wherein the total length difference between any two input vectors in the redistributed data domains is less than a first threshold.

[0089] Before executing S202 above, the management resource can obtain the total length of the input vectors in each of the M data parallel domains and determine whether the total length of the input vectors in any two data parallel domains is greater than or equal to a first threshold. If the difference in the total length of the input vectors in any two data parallel domains is greater than or equal to the first threshold, the management resource issues redistribution instructions to each of the M data parallel domains. These redistribution instructions instruct the input vectors in the data parallel domains to be redistributed. After receiving the redistribution instructions, the M data parallel domains redistribute the input vectors through parallel communication between them. Thus, when the difference in the total length of the input vectors in any two data parallel domains is small, input vector redistribution is avoided, thereby accelerating subsequent multimodal model training or inference tasks.

[0090] In S202 above, the M data parallel domains can be configured as follows: Figure 4 The steps shown below achieve the redistribution of input vectors in M ​​parallel data domains.

[0091] S401, the management resources determine the target data parallel domain corresponding to each input vector in the M data parallel domains based on the length of the input vectors in the M data parallel domains.

[0092] In the above S401, for training or inference scenarios, resource management can determine the target data parallel domain corresponding to each input vector in the M data parallel domains through any one of the following implementation methods B1, B2, and B3.

[0093] Implementation method B1 includes, for example: Figure 5 The following steps are shown.

[0094] S501, the management resources determine the ascending order number of each input vector in each of the S1 data parallel domains based on the length of each input vector in each of the S1 data parallel domains.

[0095] In the above S501, when M is even, S1 is (M / 2); when M is odd, S1 is... in, Indicates rounding up. This indicates rounding down to the nearest integer.

[0096] In S501 above, the management resources sort the input vectors in each of the S1 data parallel domains individually. For one of the S1 data parallel domains, the smaller the length of the input vector in that data parallel domain, the smaller the ascending order number of the input vector; the larger the length of the input vector in that data parallel domain, the larger the ascending order number of the input vector.

[0097] S502, the management resources determine the descending order of each input vector in each of the remaining S2 data parallel domains based on the length of each input vector in each of the remaining S2 data parallel domains.

[0098] In the above S502, when M is even, S2 is (M / 2); when M is odd, if S1 is... Then S2 is If S1 is Then S2 is

[0099] In S502 above, the management resources sort the input vectors in each of the remaining S2 data parallel domains individually. For one of the remaining S2 data parallel domains, if the length of the input vector in that data parallel domain is larger, the descending order number corresponding to that input vector is smaller; if the length of the input vector in that data parallel domain is smaller, the descending order number corresponding to that input vector is larger.

[0100] S503, the management resources determine the target data parallel domain corresponding to each input vector in the M data parallel domains based on the ascending order of the input vectors in the S1 data parallel domains and the descending order of the input vectors in the remaining S2 data parallel domains.

[0101] Implementation B1 provides a method for determining the target data parallel domain corresponding to the input vector in the data parallel domain, and the total length of the input vectors in each target data parallel domain determined by Implementation B1 is relatively similar.

[0102] In the above S503, the management resources can be repeatedly executed as follows: Figure 6 The steps shown are repeated until each input vector in each data parallel domain corresponds to the target data parallel domain.

[0103] S601, the management resources select the input vector with the smallest index in ascending order from S1 parallel data domains and the input vector with the smallest index in descending order from the remaining S2 parallel data domains respectively, to obtain M input vectors.

[0104] In the above S601, the management resources can also select the input vector with the largest ascending index from the S1 parallel data domains and the input vector with the largest descending index from the remaining S2 parallel data domains to obtain M input vectors, which is not limited here.

[0105] S602, the management resources determine one of the M data parallel domains as the target data parallel domain corresponding to the M input vectors.

[0106] In the above S602, the management resource selects one data parallel domain from the M data parallel domains in turn each time, and uses the selected data parallel domain as the target data parallel domain corresponding to the M input vectors.

[0107] For example, managing resources in determining such Figure 3 After determining the input vectors in the data parallel domain shown, the ascending order of the four input vectors in data parallel domain 1 can be identified as follows:

[0108] <Input vector 4, 1>

[0109] <Input vector 3, 2>

[0110] <Input vectors 2, 3>

[0111] <Input vector 1, 4>

[0112] Resource management can also determine the descending sort order of the four input vectors in data parallel domain 2, as follows:

[0113] <Input vector 5, 1>

[0114] <Input vector 6, 2>

[0115] <Input vector 7, 3>

[0116] <Input vector 8, 4>

[0117] The management resource first selects the input vector 4 with the smallest ascending index from data parallel domain 1 and the input vector 5 with the smallest descending index from data parallel domain 2, and sets the target data parallel domain of input vector 4 and input vector 5 as data parallel domain 1.

[0118] The management resources then select the input vector 3 with the smallest ascending index from data parallel domain 1 and the input vector 6 with the smallest descending index from data parallel domain 2, and set the target data parallel domain for input vector 3 and input vector 6 as data parallel domain 2.

[0119] The management resources then select the input vector 2 with the smallest ascending index from data parallel domain 1 and the input vector 7 with the smallest descending index from data parallel domain 2, and set the target data parallel domain of input vector 2 and input vector 7 as data parallel domain 1.

[0120] Finally, the management resources select the input vector 1 with the smallest ascending index from data parallel domain 1 and the input vector 8 with the smallest descending index from data parallel domain 2, and set the target data parallel domain for input vector 1 and input vector 8 as data parallel domain 2.

[0121] Managing resources can determine the target data parallel domain corresponding to each input vector in data parallel domain 1 and data parallel domain 2, as shown in Table 1.

[0122] Table 1.

[0123] Input vector Target data parallel domain Input vector 1 Data Parallel Domain 2 Input vector 2 Data Parallel Domain 1 Input vector 3 Data Parallel Domain 2 Input vector 4 Data Parallel Domain 1 Input vector 5 Data Parallel Domain 1 Input vector 6 Data Parallel Domain 2 Input vector 7 Data Parallel Domain 1 Input vector 8 Data Parallel Domain 2

[0124] In implementation method B2, for each of the M data parallel domains, the management resources execute as follows: Figure 7 The following steps are shown:

[0125] S701, the management resource determines the sorting number of each input vector in a data parallel domain based on the length of each input vector in the data parallel domain.

[0126] In S701 above, the management resource can determine the ascending or descending sorting number of each input vector in the data parallel domain based on the length of each input vector in the data parallel domain, without any limitation here.

[0127] S702, the management resources determine the target data parallel domain corresponding to each input vector in the data parallel domain based on the sorting number of each input vector in the data parallel domain.

[0128] Implementation B2 provides a method for determining the target data parallel domain corresponding to the input vector in the data parallel domain, and the total length of the input vectors in each target data parallel domain determined by Implementation B2 is relatively similar.

[0129] In the above S702, the management resources can be repeatedly executed as follows: Figure 8 The following steps are shown until each input vector in the data parallel domain corresponds to the target data parallel domain.

[0130] S801, the management resources select the M input vectors with the smallest sorting index from the parallel data domain.

[0131] In the above S801, the management resources can also select the M input vectors with the largest sorting index from the data parallel domain, which is not limited here.

[0132] S802, the management resources determine the target data parallel domains corresponding to the M input vectors based on the M data parallel domains.

[0133] In the above S802, the management resources can make a one-to-one correspondence between the M data parallel domains and the M input vectors, so as to determine the target data parallel domains corresponding to the M input vectors respectively.

[0134] Implementation method B3 includes, for example: Figure 9 The following steps are shown:

[0135] S901, manage resources to determine M input vector groups from input vectors in M ​​parallel data domains.

[0136] In S901 above, each input vector group includes (N / M) encoded vectors, and the difference between the total length of the (N / M) input vectors in each input vector group and the average length corresponding to the data parallel domain is less than the second threshold. Thus, the total length of the input vectors in each input vector group is approximately equal.

[0137] The average length of the data parallel domain is determined based on the N input vectors and the number of data parallel domains M. Specifically, it is determined as follows: the average length of the data parallel domain is obtained by dividing the sum of the lengths of the N input vectors by M.

[0138] S902, the management resources determine the target data parallel domains corresponding to the M input vector groups based on the M data parallel domains, wherein the target data parallel domains corresponding to the input vectors in each input vector group are the same.

[0139] In the above S902, the management resources can make a one-to-one correspondence between the M data parallel domains and the M input vector groups, thereby determining the target data parallel domains corresponding to the M input vector groups respectively.

[0140] Implementation B3 provides a method for determining the target data parallel domain corresponding to the input vector in the data parallel domain, and the total length of the input vectors in each target data parallel domain determined by Implementation B3 is relatively similar.

[0141] In the above S401, for the inference scenario, the management resources can also determine the target data parallel domain corresponding to each input vector in the M data parallel domains through implementation method C.

[0142] Implementation method C includes, for example: Figure 10 The following steps are shown:

[0143] S1001, the management resources sort the input vectors in the M parallel data domains as a whole according to the length of each input vector in the M parallel data domains, and obtain the sorting number of each input vector in the M parallel data domains.

[0144] In the embodiments of this application, the sorting index of each input vector in the M parallel data domains can be either an ascending sorting index or a descending sorting index, and is not limited here.

[0145] S1002, the management resource repeatedly executes the following steps until each input vector in the M data parallel domains corresponds to the target data parallel domain: the management resource selects the input vectors with the smallest sorting number (N / M) from the input vectors in the M data parallel domains, and selects one data parallel domain from the M data parallel domains as the target data parallel domain corresponding to the input vectors with the smallest sorting number (N / M).

[0146] In S1002 above, the management resources can also select the input vectors with the largest sorting index (N / M) from the input vectors in the M parallel data domains, without any limitation.

[0147] Implementation C provides a method for determining the target data parallel domain corresponding to an input vector in the data parallel domain. Implementation C allows input vectors of similar length to correspond to the same target data parallel domain. In inference scenarios, Implementation C enables multimodal models to output inference results corresponding to shorter input vectors earlier, which is beneficial for improving the user experience in inference scenarios.

[0148] S402, M parallel data domains redistribute the input vectors in the M parallel data domains according to the target parallel data domain corresponding to each input vector in the M parallel data domains through parallel communication between the M parallel data domains.

[0149] In S402 above, the M data parallel domains send each input vector in the M data parallel domains to the target data parallel domain corresponding to each input vector in parallel through parallel communication between the M data parallel domains, thereby realizing the redistribution of each input vector in the M data parallel domains.

[0150] In the above S402, the parallel communication between the M data parallel domains can be implemented by any one of the following implementation methods D1 or D2.

[0151] In implementation method D1, parallel communication between M parallel data domains can be achieved using the all-toall operator in the set communication library. This allows full utilization of the existing set communication library without introducing additional communication operators to achieve parallel communication between the M parallel data domains.

[0152] In implementation method D2, when the all-to-all operator in the collection communication library cannot be used, other communication operators can be introduced to achieve parallel communication between M data parallel domains.

[0153] In S402 above, for at least one input vector in one of the M data parallel domains, where the target data parallel domain corresponding to the at least one input vector is the same, the M data parallel domains perform the following steps: the M data parallel domains send the at least one input vector to the target data parallel domain corresponding to the at least one input vector through communication between the data parallel domain and the target data parallel domain corresponding to the at least one input vector.

[0154] It should be understood that after the M data parallel domains send at least one input vector to the target data parallel domain corresponding to at least one input vector, the target data parallel domain will no longer store at least one input vector. Instead, the target data parallel domain corresponding to at least one input vector will store the at least one input vector.

[0155] For example, after the management resources determine the target data parallel domains corresponding to each input vector in data parallel domain 1 and data parallel domain 2 by executing the steps in S401 (see Table 1), the M data parallel domains can execute the steps in S402 to redistribute each input vector in data parallel domain 1 and data parallel domain 2. The redistributed input vectors in data parallel domain 1 and data parallel domain 2 are as follows: Figure 11 As shown.

[0156] exist Figure 4 In the steps shown, the management resource first determines the target data parallel domain corresponding to each input vector in the M data parallel domains. Then, the M data parallel domains send each input vector to its corresponding target data parallel domain, accurately achieving the redistribution of each input vector. During this process, each of the M data parallel domains does not need to wait for the redistribution of input vectors in other data parallel domains to complete; instead, it directly achieves the redistribution of input vectors in the M data parallel domains quickly through parallel communication between them, which helps improve redistribution efficiency.

[0157] S203, the first data parallel domain uses a multimodal model deployed on the first data parallel domain to calculate the redistributed input vector in the first data parallel domain.

[0158] In S204 above, a complete and identical multimodal model is deployed on each of the M data parallel domains, and each data parallel domain executes the steps in S203 above simultaneously.

[0159] In the training scenario, the first data parallel domain can use the redistributed input vectors in the first data parallel domain to train the multimodal model deployed on the first data parallel domain, so as to achieve the purpose of training the multimodal model.

[0160] In the inference scenario, the first data parallel domain can use the redistributed input vectors in the first data parallel domain to perform inference on the multimodal model, so as to achieve the purpose of using the multimodal model for inference.

[0161] exist Figure 2 In the steps shown, each of the M data parallel domains does not need to wait for the redistribution of input vectors in other data parallel domains to complete. The M data parallel domains directly and quickly redistribute the input vectors in the M data parallel domains through parallel communication, which helps to improve redistribution efficiency. Furthermore, since the total length difference of the redistributed input vectors in each data parallel domain is small, the load of each data parallel domain can be balanced when calculating the redistributed input vectors in each data parallel domain.

[0162] Based on the above embodiments, this application also provides an input data processing device for a multimodal model, which is applied to... Figure 1A The AI ​​cluster 102 shown is used to implement Figure 2 The illustrated embodiment provides a method for processing input data for a multimodal model. See also... Figure 12 As shown, the input data processing device for the multimodal model includes: an acquisition module 1201, a redistribution module 1202, a calculation module 1203, and a determination module 1204. Wherein:

[0163] The acquisition module 1201 is used to acquire multiple input vectors corresponding to the first data parallel domain among M data parallel domains; the first data parallel domain is any one of the M data parallel domains, and the M data parallel domains correspond to N input vectors; M and N are both greater than 1.

[0164] The redistribution module 1202 is used to redistribute the input vectors in the M data parallel domains through parallel communication between the M data parallel domains; the difference in the total length of the input vectors in any two data parallel domains after redistribution is less than a first threshold.

[0165] The computation module 1203 is used to compute the redistributed input vector in the first data parallel domain using a multimodal model deployed on the first data parallel domain. Each of the M data parallel domains deploys a complete and identical multimodal model.

[0166] In one possible implementation, the redistribution module 1202 is specifically used to: redistribute the input vectors in the M data parallel domains through parallel communication between the M data parallel domains when the total length difference between any two data parallel domains is greater than or equal to a first threshold.

[0167] In one possible implementation, the determining module 1204 is used to determine the target data parallel domain corresponding to each input vector in the M data parallel domains based on the length of the input vectors in the M data parallel domains;

[0168] The redistribution module 1202 is specifically used to: redistribute the input vectors in the M data parallel domains according to the target data parallel domains corresponding to each input vector in the M data parallel domains through parallel communication between the M data parallel domains.

[0169] In one possible implementation, the redistribution module 1202 is specifically used to: send each input vector in the M data parallel domains in parallel to the target data parallel domain corresponding to each input vector through parallel communication between the M data parallel domains, thereby realizing the redistribution of each input vector in the M data parallel domains.

[0170] In one possible implementation, the determining module 1204 is specifically used to: determine the ascending order index of each input vector in each of the S1 data parallel domains based on the length of each input vector in each of the S1 data parallel domains; when M is even, S1 is (M / 2); when M is odd, S1 is... Based on the lengths of the input vectors in the remaining S2 data parallel domains, determine the descending order of the input vectors in each of the remaining S2 data parallel domains; when M is even, S2 is (M / 2); when M is odd, S2 is... Based on the ascending order of the input vectors in the S1 data parallel domains and the descending order of the input vectors in the remaining S2 data parallel domains, determine the target data parallel domain corresponding to each input vector in the M data parallel domains.

[0171] In one possible implementation, the determining module 1204 is specifically used to: for one of the M data parallel domains, perform the following steps: determine the sorting index of each input vector in the data parallel domain according to the length of each input vector in the data parallel domain; determine the target data parallel domain corresponding to each input vector in the data parallel domain according to the sorting index of each input vector in the data parallel domain.

[0172] In one possible implementation, the determining module 1204 is specifically used to: determine M input vector groups from the input vectors in M ​​data parallel domains; each input vector group includes (N / M) encoded vectors, and the difference between the total length of the (N / M) input vectors in each input vector group and the average length corresponding to the data parallel domain is less than a second threshold; the average length corresponding to the data parallel domain is determined based on the N input vectors; and based on the M data parallel domains, determine the target data parallel domains corresponding to the M input vector groups respectively, wherein the target data parallel domains corresponding to the input vectors in each input vector group are the same.

[0173] In one possible implementation, the determining module 1204 is specifically used to: sort the input vectors in the M data parallel domains as a whole according to the length of each input vector in the M data parallel domains, and obtain the sorting index of each input vector in the M data parallel domains; repeat the following steps until each input vector in the M data parallel domains corresponds to the target data parallel domain: select the input vectors with the smallest sorting index (N / M) from the input vectors in the M data parallel domains, and select a data parallel domain from the M data parallel domains as the target data parallel domain corresponding to the input vectors with the smallest sorting index (N / M).

[0174] In the embodiments of this application, the acquisition module, redistribution module, calculation module, and determination module can all be implemented in software or in hardware. For example, the implementation of the acquisition module will be described below. Similarly, the implementation of the redistribution module, calculation module, and determination module can refer to the implementation of the acquisition module.

[0175] As an example of a software functional unit, a module can include code running on a computing instance. A computing instance can include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned computing instance can be one or more. For example, a module can include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code can be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code can be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region can include multiple AZs.

[0176] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0177] As an example of a hardware functional unit, an acquisition module may include at least one computing device, such as a server. Alternatively, an acquisition module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0178] The acquisition module includes multiple computing devices that can be distributed within the same region or in different regions. Similarly, the acquisition module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the acquisition module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0179] It should be noted that, in other embodiments, the acquisition module is used to acquire multiple input vectors corresponding to the first data parallel domain among M data parallel domains; the first data parallel domain is any one of the M data parallel domains, and the M data parallel domains correspond to N input vectors; M and N are both greater than 1; the redistribution module is used to redistribute the input vectors in the M data parallel domains through parallel communication between the M data parallel domains; the total length difference between any two input vectors in the redistributed data parallel domains is less than a first threshold; the calculation module is used to calculate the redistributed input vectors in the first data parallel domain using a multimodal model deployed on the first data parallel domain, and a complete and identical multimodal model is deployed on each of the M data parallel domains.

[0180] The steps implemented by the acquisition module, redistribution module, calculation module, and determination module can be specified as needed. By implementing different steps in the input data processing method of the multimodal model through the acquisition module, redistribution module, calculation module, and determination module, the full functions of the input data processing device of the multimodal model can be realized.

[0181] This application also provides a computing device, which includes a processor and a memory, wherein the memory stores computer-executable instructions; the processor is used to execute the computer-executable instructions stored in the memory, and the processor can execute a multimodal model input data processing method.

[0182] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing system or stored on any usable medium. When the computer program product runs on at least one computing system, it causes the at least one computing system to execute a multimodal model input data processing method.

[0183] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing system can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing system to execute an input data processing method for a multimodal model.

[0184] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

[0185] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0186] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0187] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more flowcharts and / or one or more block diagrams.

[0188] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more flowcharts and / or one or more block diagrams.

[0189] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A method for processing input data of a multimodal model, characterized in that, include: Obtain multiple input vectors corresponding to the first data parallel domain among M data parallel domains; The first data parallel domain is any one of the M data parallel domains, and the M data parallel domains correspond to N input vectors; M and N are both greater than 1. The redistribution of input vectors in the M data parallel domains is achieved through parallel communication between the M data parallel domains; The difference in the total length of the input vectors in any two parallel data domains after redistribution is less than the first threshold. The multimodal model deployed on the first data parallel domain is used to calculate the redistributed input vector in the first data parallel domain. Each of the M data parallel domains deploys a complete and identical multimodal model.

2. The method as described in claim 1, characterized in that, The redistribution of input vectors in the M data parallel domains through parallel communication includes: If the difference in the total length of the input vectors in any two of the M data parallel domains is greater than or equal to the first threshold, the redistribution of the input vectors in the M data parallel domains is achieved through parallel communication between the M data parallel domains.

3. The method as described in any one of claims 1-2, characterized in that, Before the redistribution of input vectors in the M data parallel domains through parallel communication between the M data parallel domains, the method further includes: Based on the lengths of the input vectors in the M data parallel domains, determine the target data parallel domain corresponding to each input vector in the M data parallel domains; The redistribution of input vectors in the M data parallel domains through parallel communication includes: Based on the target data parallel domain corresponding to each input vector in the M data parallel domains, the redistribution of each input vector in the M data parallel domains is achieved through parallel communication between the M data parallel domains.

4. The method as described in claim 3, characterized in that, The step of redistributing the input vectors in the M data parallel domains according to the target data parallel domains corresponding to each input vector in the M data parallel domains, through parallel communication between the M data parallel domains, includes: By means of parallel communication between the M data parallel domains, each input vector in the M data parallel domains is sent in parallel to the target data parallel domain corresponding to each input vector, thereby realizing the redistribution of each input vector in the M data parallel domains.

5. The method as described in any one of claims 3-4, characterized in that, The step of determining the target data parallel domain corresponding to each input vector in the M data parallel domains based on the length of the input vectors in the M data parallel domains includes: Based on the length of each input vector in the S1 parallel data domains, determine the ascending order index of each input vector in each of the S1 parallel data domains; when M is even, S1 is (M / 2); when M is odd, S1 is... or Based on the lengths of the input vectors in the remaining S2 data parallel domains, determine the descending order of the input vectors in each of the remaining S2 data parallel domains; when M is even, S2 is (M / 2); when M is odd, S2 is... or Based on the ascending order of the input vectors in the S1 data parallel domains and the descending order of the input vectors in the remaining S2 data parallel domains, the target data parallel domain corresponding to each input vector in the M data parallel domains is determined.

6. The method as described in any one of claims 3-4, characterized in that, The step of determining the target data parallel domain corresponding to each input vector in the M data parallel domains based on the length of the input vectors in the M data parallel domains includes: For one of the M data parallel domains, perform the following steps: Based on the length of each input vector in the data parallel domain, determine the sorting number of each input vector in the data parallel domain; Based on the sorting index of each input vector in the data parallel domain, the target data parallel domain corresponding to each input vector in the data parallel domain is determined.

7. The method as described in any one of claims 3-4, characterized in that, The step of determining the target data parallel domain corresponding to each input vector in the M data parallel domains based on the length of the input vectors in the M data parallel domains includes: M input vector groups are determined from the input vectors in the M data parallel domains; each input vector group includes (N / M) encoded vectors, and the difference between the total length of the (N / M) input vectors in each input vector group and the average length corresponding to the data parallel domain is less than a second threshold; the average length corresponding to the data parallel domain is determined based on the N input vectors; Based on the M data parallel domains, the target data parallel domains corresponding to the M input vector groups are determined respectively, wherein the target data parallel domains corresponding to the input vectors in each input vector group are the same.

8. An input data processing device for a multimodal model, characterized in that, include: The acquisition module is used to acquire multiple input vectors corresponding to the first data parallel domain among M data parallel domains; The first data parallel domain is any one of the M data parallel domains, and the M data parallel domains correspond to N input vectors; M and N are both greater than 1. The redistribution module is used to redistribute the input vectors in the M data parallel domains through parallel communication between the M data parallel domains; The difference in the total length of the input vectors in any two parallel data domains after redistribution is less than the first threshold. The computation module is used to compute the redistributed input vector in the first data parallel domain using a multimodal model deployed on the first data parallel domain, wherein a complete and identical multimodal model is deployed on each of the M data parallel domains.

9. The apparatus as claimed in claim 8, characterized in that, The redistribution module is specifically used for: If the difference in the total length of the input vectors in any two of the M data parallel domains is greater than or equal to the first threshold, the redistribution of the input vectors in the M data parallel domains is achieved through parallel communication between the M data parallel domains.

10. The apparatus as described in any one of claims 8-9, characterized in that, The device also includes a determining module; The determining module is used to determine the target data parallel domain corresponding to each input vector in the M data parallel domains based on the length of the input vectors in the M data parallel domains; The redistribution module is specifically used for: Based on the target data parallel domain corresponding to each input vector in the M data parallel domains, the redistribution of each input vector in the M data parallel domains is achieved through parallel communication between the M data parallel domains.

11. The apparatus as claimed in claim 10, characterized in that, The redistribution module is specifically used for: By means of parallel communication between the M data parallel domains, each input vector in the M data parallel domains is sent in parallel to the target data parallel domain corresponding to each input vector, thereby realizing the redistribution of each input vector in the M data parallel domains.

12. A computing device, characterized in that, The device includes a processor and a memory; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to cause the computing device to perform the method as described in any one of claims 1-7.

13. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device, the computing device performs the method as described in any one of claims 1-7.

Images