A data processing method and apparatus

Abstract

The application provides a data processing method and device, the method comprising: an i-th computing device splitting i-th first vector data according to a first dimension to obtain M first sub-vector data; after performing second dimension calculation on the m-th first sub-vector data, sending the second dimension calculation result corresponding to the m-th first sub-vector data to other N-1 computing devices, receiving the second dimension calculation result corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices, and performing second dimension calculation on the m+1-th first sub-vector data; finally, generating the i-th vector result according to the second dimension calculation result corresponding to each of the M first sub-vector data and the second dimension calculation result corresponding to each of the M second sub-vector data sent by each of the other N-1 computing devices. The above process can reduce the time length of model training or reasoning.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a data processing method and apparatus. Background Technology

[0002] In certain model training or inference scenarios, after one computing device in a computing cluster performs calculations on vector data and obtains the result, it needs to synchronize this result to other computing devices in the cluster. During this process, other computing devices can only obtain the result after the first device has completed its calculations and obtained the result. This leads to a prolonged model training or inference time. Reducing the training or inference time is a pressing technical problem that needs to be solved. Summary of the Invention

[0003] This application provides a data processing method and apparatus for reducing the time required for model training or inference.

[0004] In a first aspect, embodiments of this application provide a data processing method, which can be executed by the i-th computing device among N computing devices, where i is any integer between [1, N] and N is an integer greater than 1. The method specifically includes the following steps: the i-th computing device obtains the i-th first vector data processed by the i-th computing device from the vector data processed by the N computing devices, wherein the vector data includes data of multiple dimensions, and the i-th first vector data is the i-th part of the data after dividing the data of multiple dimensions into N parts according to the first dimension; the i-th computing device divides the i-th first vector data according to the first dimension to obtain M first sub-vector data; M is an integer greater than 1; after performing the calculation of the second dimension on the m-th first sub-vector data, the i-th computing device sends the calculation result of the second dimension corresponding to the m-th first sub-vector data to the other N-1 computing devices, and receives the m-th second sub-vector data sent by each of the other N-1 computing devices. The calculation result of the second dimension is performed on the (m+1)th first sub-vector data, and the calculation result of the second dimension is performed on the first sub-vector data until the calculation result of the second dimension corresponding to the Mth first sub-vector data is sent to the other N-1 computing devices, and the calculation result of the second dimension corresponding to the Mth second sub-vector data sent by each of the other N-1 computing devices is received, where m is an integer between [1, M-1]. The i-th computing device generates the i-th vector result based on the calculation results of the second dimension corresponding to the M first sub-vector data and the calculation results of the second dimension corresponding to the M second sub-vector data sent by each of the other N-1 computing devices. The i-th vector result is the calculation result of the i-th first vector data in the second dimension.

[0005] In the above method, the i-th computing device does not perform the calculation of the second dimension on the i-th first vector data and then send the calculation result of the second dimension corresponding to the i-th first vector data to other computing devices. Instead, it first divides the i-th first vector data according to the first dimension to obtain M first sub-vector data. After performing the calculation of the second dimension on the m-th first sub-vector data among the M first sub-vector data, it sends the calculation result of the second dimension corresponding to the m-th first sub-vector data to the other N-1 computing devices, and receives the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices. At the same time, it performs the calculation of the second dimension on the (m+1)-th first sub-vector data. As can be seen from the above process, the i-th computing device simultaneously executes the calculation result of the second dimension corresponding to the m-th first sub-vector data, receives the calculation result of the second dimension corresponding to the m-th second sub-vector data from each of the other N-1 computing devices, and performs the calculation of the second dimension on the m+1-th first sub-vector data. The above process can save data processing time in model training or inference, thereby reducing the duration of model training or inference.

[0006] In one possible design, the above method may further include: if there are X dimensions that need to be calculated among multiple dimensions, and if the second dimension is not the last dimension that needs to be calculated, then the generated i-th vector result is split in the second dimension so that the split data can be calculated in the third dimension; if the second dimension is the last dimension that needs to be calculated, then the i-th vector result is taken as the i-th calculation result of the vector data in the X dimensions, where X is an integer greater than 1.

[0007] In one possible design, the i-th computing device sends the calculation result corresponding to the m-th first sub-vector data to the other N-1 computing devices, and receives the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices. This can include: the i-th computing device sending the calculation result corresponding to the m-th first sub-vector data to the other N-1 computing devices through parallel communication between the N computing devices, and receiving the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices.

[0008] In one possible design, the i-th computing device sends the calculation result of the second dimension corresponding to the m-th first sub-vector data to the other N-1 computing devices, and receives the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices. This can include: the i-th computing device sending the j-th calculation result fragment of the N calculation result fragments corresponding to the m-th first sub-vector data to the j-th computing device in the computing device cluster, and receiving the i-th calculation result fragment of the N calculation result fragments corresponding to the m-th second sub-vector data sent by the j-th computing device; j is an integer taking values ​​from [1, N], and j is not equal to i; the m-th... The N computational result fragments corresponding to a sub-vector data are obtained by the i-th computing device, which divides the computational result of the second dimension corresponding to the m-th first sub-vector data in the second dimension according to the number N of computing devices in the computing device cluster; the N computational result fragments corresponding to the m-th second sub-vector data are obtained by the j-th computing device, which divides the computational result of the second dimension corresponding to the m-th second sub-vector data in the second dimension according to the number N of computing devices in the computing device cluster; this design provides a specific method for the i-th computing device to send the computational result of the second dimension corresponding to the m-th first sub-vector data and to receive the computational result of the second dimension corresponding to the m-th second sub-vector data.

[0009] The i-th computing device generates the i-th vector result based on the calculation results of the second dimension corresponding to the M first sub-vector data, and the calculation results of the second dimension corresponding to the M second sub-vector data sent by each of the other N-1 computing devices. This can include: the i-th computing device generating the i-th vector result based on the i-th calculation result fragment corresponding to the M first sub-vector data, and the i-th calculation result fragment corresponding to the M second sub-vector data sent by the j-th computing device. This design provides a specific method for generating the i-th vector result.

[0010] In one possible design, the i-th computing device generates the i-th vector result based on the i-th computation result fragments corresponding to the M first sub-vector data and the i-th computation result fragments corresponding to the M second sub-vector data sent by the j-th computing device. This can include: the i-th computing device sorting the i-th computation result fragments corresponding to the M first sub-vector data and the i-th computation result fragments corresponding to the M second sub-vector data sent by the j-th computing device according to the numbers of the N computing devices, obtaining a first sorting result; the i-th computing device concatenating the first sorting result along the first dimension to obtain the i-th vector result. This design provides a specific method for generating the i-th vector result.

[0011] In one possible design, the vector data is the data corresponding to the multimodal model performing training or inference tasks, the calculation is an attention calculation, and the calculation of the X dimensions of data does not affect each other.

[0012] Secondly, embodiments of this application also provide a data processing apparatus, which includes methods for performing the methods provided in the first aspect above. The data processing apparatus includes: an acquisition module, a segmentation module, a synchronous execution module, and a generation module.

[0013] The acquisition module is used to acquire the i-th first vector data processed by the i-th computing device from the vector data processed by N computing devices. The vector data includes data of multiple dimensions. The i-th first vector data is the i-th part of the data after the data of multiple dimensions is divided into N parts according to the first dimension; i is any integer between [1, N] and N is an integer greater than 1.

[0014] The segmentation module is used to segment the i-th first vector data according to the first dimension to obtain M first sub-vector data; M is an integer greater than 1;

[0015] The synchronous execution module is used to, after performing the calculation of the second dimension on the m-th first sub-vector data, send the calculation result of the second dimension corresponding to the m-th first sub-vector data to the other N-1 computing devices, and receive the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices. At the same time, it performs the calculation of the second dimension on the (m+1)-th first sub-vector data until the calculation result of the second dimension corresponding to the M-th first sub-vector data is sent to the other N-1 computing devices, and receives the calculation result of the second dimension corresponding to the M-th second sub-vector data sent by each of the other N-1 computing devices, where m is an integer taken from [1, M-1].

[0016] The generation module is used to generate the i-th vector result based on the calculation results of the second dimension corresponding to the M first sub-vector data respectively, and the calculation results of the second dimension corresponding to the M second sub-data vectors sent by each of the other N-1 computing devices. The i-th vector result is the calculation result of the i-th first vector data in the second dimension.

[0017] In one possible design, the synchronous execution module is also used for: if there are X dimensions that need to be calculated among multiple dimensions, and if the second dimension is not the last dimension that needs to be calculated, then the generated i-th vector result is split in the second dimension so that the split data can be calculated in the third dimension; if the second dimension is the last dimension that needs to be calculated, then the i-th vector result is used as the i-th calculation result of the vector data in the X dimensions, where X is an integer greater than 1.

[0018] In one possible design, the synchronous execution module is specifically used to: send the calculation result corresponding to the m-th first sub-vector data to the other N-1 computing devices through parallel communication between N computing devices, and receive the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices.

[0019] In one possible design, the synchronous execution module is specifically used to: send the j-th calculation result fragment from the N calculation result fragments corresponding to the m-th first sub-vector data to the j-th computing device in the computing device cluster, and receive the i-th calculation result fragment from the N calculation result fragments corresponding to the m-th second sub-vector data sent by the j-th computing device; j is an integer between [1, N], and j is not equal to i; the N calculation result fragments corresponding to the m-th first sub-vector data are obtained by the i-th computing device by splitting the calculation result of the second dimension corresponding to the m-th first sub-vector data in the second dimension according to the number N of computing devices in the computing device cluster; the N calculation result fragments corresponding to the m-th second sub-vector data are obtained by the j-th computing device by splitting the calculation result of the second dimension corresponding to the m-th second sub-vector data in the second dimension according to the number N of computing devices in the computing device cluster.

[0020] The generation module is specifically used to generate the i-th vector result based on the i-th calculation result fragment corresponding to each of the M first sub-vector data and the i-th calculation result fragment corresponding to each of the M second sub-vector data sent by the j-th computing device.

[0021] In one possible design, the generation module is specifically used to: sort the i-th calculation result fragments corresponding to the M first sub-vector data and the i-th calculation result fragments corresponding to the M second sub-vector data sent by the j-th computing device according to the numbers of the N computing devices, and obtain a first sorting result; and concatenate the first sorting results in the first dimension to obtain the i-th vector result.

[0022] In one possible design, the vector data is the data corresponding to the multimodal model performing training or inference tasks, the calculation is an attention calculation, and the calculation of the X dimensions of data does not affect each other.

[0023] Thirdly, this application also provides a computing device, including a processor and a memory, wherein the memory stores computer-executable instructions; the processor is used to execute the computer-executable instructions stored in the memory, and the processor performs the method provided in the first aspect above.

[0024] Fourthly, this application also provides a computer program product containing instructions that, when executed by a computing device, cause the computing device to perform the method provided in the first aspect above.

[0025] Fifthly, this application also provides a computer-readable storage medium including computer program instructions, which, when executed by a computing device, cause the computing device to perform the method provided in the first aspect above. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of a video generation process provided in an embodiment of this application;

[0027] Figure 2 A system architecture diagram provided for an embodiment of this application;

[0028] Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of this application;

[0029] Figure 4 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0030] Figure 5 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0031] Figure 6 A flowchart illustrating a method for generating the i-th vector result provided in an embodiment of this application;

[0032] Figure 7 A flowchart illustrating a data processing method provided in an embodiment of this application;

[0033] Figure 8 This is a schematic diagram of the structure of a data processing device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. The terms "first," "second," and corresponding reference numerals in the specification, claims, and drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of units is not necessarily limited to those units, but may include other units not explicitly listed or inherent to these processes, methods, products, or apparatuses.

[0035] The following explanations of some terms used in the embodiments of this application are provided to facilitate understanding by those skilled in the art.

[0036] A multimodal model is a model that can understand and process multiple types of data. These multiple types of data include, but are not limited to, text, images, audio, and video.

[0037] Data parallelism is a widely used technique in distributed systems that aims to improve computational efficiency and speed by dividing a dataset into multiple subsets and processing these subsets in parallel on different computing devices. In deep learning and large-scale machine learning training, data parallelism can significantly accelerate the training speed of artificial intelligence models and improve model performance.

[0038] Taking the generation of video scenes using Open-Sora as an example, the video generation process can be referenced. Figure 1 As shown.

[0039] The prompt information acquisition unit 101 is used to acquire prompt information, such as "Generate a video of playing basketball". After acquiring the prompt information, the prompt information acquisition unit 101 sends the prompt information to the text encoder 102. After acquiring the prompt information, the text encoder 102 encodes the prompt information to obtain the vector corresponding to the prompt information, and sends the vector corresponding to the prompt information to the spatial temporal diffusion transformer (STDIT) 104.

[0040] The random noise generation unit 103 can generate a random Gaussian noise vector and send the random Gaussian noise vector to the spatiotemporal diffusion transformer 104. The random Gaussian noise vector is used to generate a video corresponding to the prompt information.

[0041] After acquiring the vector corresponding to the prompt information and the random Gaussian noise vector, the spatiotemporal diffusion transformer 104 processes the vector corresponding to the prompt information and the random Gaussian noise vector to obtain the encoded video, and sends the encoded video to the variational autoencoder (VAE) 105. Specifically, the encoded video is generated by the spatiotemporal diffusion transformer 104 using the vector corresponding to the prompt information and processing the random Gaussian noise vector.

[0042] After acquiring the encoded video, the variational autoencoder 105 decodes the encoded video to obtain a viewable and playable video that corresponds to the prompt information. For example, if the prompt information is "Generate a video of playing basketball", the variational autoencoder 105 can output a video of playing basketball.

[0043] In one possible implementation, the process of the spatiotemporal diffusion transformer 104 processing the vector corresponding to the prompt information and the random Gaussian noise vector may include the following three steps: spatial transformation, temporal transformation, and parallel communication. The specific process is as follows:

[0044] The random Gaussian noise vector can be represented as a 4-dimensional vector [b,t,s,hn*hd], where the first dimension b represents the number of videos to be generated; the second dimension t is the time dimension, used to represent the number of frames in a video; the third dimension s is the spatial dimension, used to represent the number of pixels in each frame; and the fourth dimension hn*hd is used for subsequent attention calculations, where hn represents the number of heads in the attention mechanism and hd represents the size of the hidden layer.

[0045] To accelerate the processing efficiency of the spatiotemporal diffusion transformer 104, N computing devices can process the random Gaussian noise vector in a data-parallel manner. Specifically, the random Gaussian noise vector [b,t,s,hn*hd] is divided into N parts in the time dimension, each part being a 4-dimensional vector [b,t / N,s,hn*hd]. Each of the N computing devices processes one part of the N parts, and different computing devices process different parts of the N parts. Taking the i-th computing device among the N computing devices as an example, where i is any integer between [1,N] and N is an integer greater than 1, the i-th computing device obtains the first data [b,t / N,s,hn*hd] from the random Gaussian noise vector [b,t,s,hn*hd]. Here, the first data is the i-th part of the random Gaussian noise vector [b,t,s,hn*hd] after dividing it into N parts in the time dimension, and / represents the division sign.

[0046] (1) The i-th computing device first performs a spatial transformation, that is, the i-th computing device uses the first data [b,t / N,s,hn*hd] and the vector corresponding to the prompt information to perform attention calculation in the spatial dimension to generate the second data [b,t / N,s,hn*hd]. Then, the i-th computing device further segments the second data [b,t / N,s,hn*hd] in the spatial dimension, that is, the i-th computing device segments the second data [b,t / N,s,hn*hd] into N parts in the spatial dimension to obtain N third data [b,t / N,s / N,hn*hd]. Among them, each of the N third data is a 4-dimensional vector [b,t / N,s / N,hn*hd].

[0047] (2) The i-th computing device performs parallel communication. That is, the i-th computing device sends N-1 third data [b,t / N,s / N,hn*hd] from the N third data it generates to the N-1 computing devices other than the i-th computing device. At the same time, it can also receive one third data [b,t / N,s / N,hn*hd] sent by each of the N-1 computing devices through parallel communication. Therefore, the i-th computing device can obtain N third data [b,t / N,s / N,hn*hd]. The N third data obtained include one third data from the N third data it generates and one third data sent by each of the N-1 computing devices. Afterwards, the i-th computing device concatenates the N third data in the time dimension to generate the fourth data [b,t,s / N,hn*hd].

[0048] (3) The i-th computing device performs a time transformation, that is, the i-th computing device uses the fourth data and the vector corresponding to the prompt information to perform attention calculation in the time dimension to generate the fifth data [b,t,s / N,hn*hd]. Then, the i-th computing device further segments the fifth data [b,t,s / N,hn*hd] in the time dimension, that is, the i-th computing device segments the fifth data [b,t,s / N,hn*hd] into N parts in the time dimension to obtain N sixth data [b,t / N,s / N,hn*hd]. Among them, each of the N sixth data is a 4-dimensional vector [b,t / N,s / N,hn*hd].

[0049] (4) The i-th computing device performs parallel communication. That is, the i-th computing device sends N-1 of its own generated N sixth data [b,t / N,s / N,hn*hd] to N-1 computing devices other than the i-th computing device through parallel communication. At the same time, it can also receive one sixth data [b,t / N,s / N,hn*hd] sent by each of the N-1 computing devices through parallel communication. Therefore, the i-th computing device can obtain N sixth data [b,t / N,s / N,hn*hd]. The N sixth data obtained include one sixth data from its own generated N sixth data and one sixth data sent by each of the N-1 computing devices. Afterwards, the i-th computing device concatenates the N sixth data in the spatial dimension to generate the seventh data [b,t / N,s,hn*hd].

[0050] Afterwards, the i-th computing device can repeat steps (1) to (4) above multiple times in order to generate a high-quality encoded video based on the vector corresponding to the prompt information and the random Gaussian noise vector, so as to facilitate the subsequent parsing of the high-quality video from the encoded video.

[0051] In steps (1) and (2) above, the i-th computing device first performs calculations in the spatial dimension to obtain the second data before it can perform subsequent parallel communication; in steps (3) and (4) above, the i-th computing device first performs calculations in the temporal dimension to obtain the fifth data before it can perform subsequent parallel communication. The above execution process will result in a long video generation time.

[0052] Combining the processes of steps (1) and (2) and steps (3) and (4) above, it can be seen that the processing of steps (1) and (2) above is similar to that of steps (3) and (4) above. The only difference is that step (1) performs attention calculation in the spatial dimension, while step (3) performs attention calculation in the temporal dimension. Based on the above similar processing processes and the similar technical problems brought about by the similar processing processes (such as causing a long video generation time), this application provides a data processing method and apparatus.

[0053] It should be understood that the data processing method and apparatus proposed in the embodiments of this application can be applied to video generation scenarios, or to other scenarios, and are not limited herein.

[0054] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0055] Figure 2 This diagram illustrates a possible system architecture for the data processing method provided in this application embodiment. The system architecture includes a terminal device 201 and a computing device cluster 202, wherein the terminal device 201 and the computing device cluster 202 can be connected via wired or wireless means.

[0056] The components of this system architecture are described below.

[0057] (1) Terminal equipment 201

[0058] Users can input a preset component M at terminal device 201, and terminal device 201 will send the preset component M to computing device cluster 202. Here, M is an integer greater than 1.

[0059] In the embodiments of this application, the terminal device 201 can provide voice and / or data connectivity to the user. For example, the terminal device is a device with wireless transceiver capabilities, which can be deployed on land, including indoors or outdoors, handheld, wearable, or vehicle-mounted; it can also be deployed on water (such as ships); and it can also be deployed in the air (such as airplanes, balloons, and satellites). The terminal device can be a mobile phone, tablet computer, computer, virtual reality (VR) terminal, augmented reality (AR) terminal, vehicle-mounted terminal, wireless terminal in self-driving, wireless terminal in remote medical care, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, wearable terminal, etc. The embodiments of this application do not limit the application scenarios.

[0060] Terminal device 201 can be fixed or mobile. It is understood that all or part of the functions of terminal device 201 in this application can also be implemented by software functions running on hardware, or by virtualization functions instantiated on a platform (e.g., a cloud platform).

[0061] (2) Computing device cluster 202

[0062] The computing device cluster 202 consists of N computing devices, where N is an integer greater than 1. Combined with... Figure 2 The computing device cluster in the middle, Figure 3 This application provides an exemplary structural diagram of a computing device 300, which may be... Figure 2 Any one of the computing devices in the cluster.

[0063] The computing device 300 includes a processor 301 and a memory 302. Optionally, the computing device 300 may also include a communication interface 304, wherein the processor 301, memory 302, and communication interface 304 can be interconnected via a communication line 303; the communication line 303 can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The communication line 303 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0064] The processor 301 includes at least an acceleration device. This acceleration device can be a graphics processing unit (GPU), a tensor processing unit (TPU), a neural network processing unit (NPU), a deep learning processing unit (DPU), etc., and is not limited thereto.

[0065] Processor 301 may also include a general-purpose processor, such as a central processing unit (CPU), etc., without limitation.

[0066] Specifically, processor 301 is used to process data access requests from outside the computing device 300 (such as other computing devices), and also to process requests generated internally by the computing device 300. In addition, processor 301 is also used to perform other data calculations or processing, such as matrix operations, without limitation.

[0067] Communication interface 304 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), wired access network, etc. For example, it can handle read data requests, write data requests, etc.

[0068] Memory 302 is used to store data and computer-executable program code. Memory 302 can exist independently and be connected to processor 301 via communication line 303. Memory 302 can also be integrated with processor 301. Specifically, memory 302 can include RAM or a hard disk. RAM refers to internal storage that directly exchanges data with processor 301. It can read and write data at any time and at a high speed, serving as temporary data storage for the operating system or other running programs. Unlike RAM, hard disk read and write data at a slower speed and is typically used for persistent data storage. In some application scenarios, processor 301 can temporarily store data in RAM. When the total amount of data in RAM reaches a certain threshold, processor 301 sends the data stored in RAM to the hard disk for persistent storage. This data can be obtained from external devices, input by the user, or generated by computing device 300, without specific limitations. Alternatively, processor 301 can read data from RAM. If a memory cache miss occurs, processor 301 reads the data from the hard disk into RAM and then reads the data from RAM again.

[0069] The memory includes at least two types of storage, such as random access memory (RAM) or read-only memory (ROM). For example, RAM can be dynamic random access memory (DRAM) or storage class memory (SCM), where DRAM can be high bandwidth memory (HBM). For read-only memory, examples include programmable read-only memory (PROM) and erasable programmable read-only memory (EPROM). In practical applications, the computing device 300 can be configured with multiple memory modules of different types. This embodiment does not limit the number or type of memory.

[0070] A hard drive can specifically be a disk or other type of storage medium, such as a solid-state drive (SSD), a hard disk drive (HDD), a shingled magnetic recording hard disk, a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, universal optical discs, Blu-ray discs, etc.), a disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and that can be accessed by a computer, but is not limited thereto.

[0071] In this embodiment of the application, each computing device in the computing device cluster 202 can obtain a preset segmentation M. Taking the i-th computing device in the computing device cluster 202 as an example, i is any integer between [1, N]. The i-th computing device obtains the i-th first vector data processed by the i-th computing device from the vector data to be processed by the computing device cluster 202. The vector data to be processed by the computing device cluster 202 includes data of multiple dimensions. The i-th first vector data is the i-th part of the data after the data of multiple dimensions is divided into N parts according to the first dimension. The first dimension can be any dimension among the multiple dimensions. Subsequently, the i-th computing device segments the i-th first vector data according to the first dimension based on the preset segmentation component M, obtaining M first sub-vector data. After performing the second dimension calculation on the m-th first sub-vector data, the i-th computing device sends the calculation result corresponding to the m-th first sub-vector data to the other N-1 computing devices, and receives the second dimension calculation result corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices. At the same time, the i-th computing device performs the second dimension calculation on the (m+1)-th first sub-vector data, until the calculation result corresponding to the M-th first sub-vector data is obtained. The result is sent to the other N-1 computing devices, and the calculation result of the second dimension corresponding to the Mth second sub-vector data sent by each of the other N-1 computing devices is received. The second dimension is a dimension other than the first dimension among multiple dimensions, and m is an integer between [1,M]. The i-th computing device generates the i-th calculation result based on the calculation results corresponding to the M first sub-vector data and the calculation results corresponding to the M second sub-vector data sent by each of the other N-1 computing devices. The i-th calculation result is the calculation result of the i-th first vector data in the second dimension.

[0072] It should be understood that, based on Figure 2 Chinese system architecture and Figure 3Modifications and variations to the architecture diagram of the computing device do not depart from the scope of protection of this application.

[0073] Combination Figure 2 The system architecture diagram shown in this application illustrates a data processing method provided in this embodiment. This method can be performed by... Figure 2 Each computing device in the computing device cluster 202 shown executes the method. Taking the i-th computing device in the computing device cluster 202 as an example, where i is any integer between [1, N], the method may include, for example, Figure 4 The following process is shown.

[0074] S401, the i-th computing device obtains the i-th first vector data processed by the i-th computing device from the vector data processed by the N computing devices.

[0075] In the above S401, the vector data to be processed by the N computing devices can be the data corresponding to the training or inference tasks of the multimodal model. The vector data includes data of multiple dimensions, of which X dimensions need to be calculated, and the calculation of the X dimensions does not affect each other, where X is an integer greater than 1.

[0076] For example, in a video generation scenario, the vector data can be a 4-dimensional vector [b, t, s, hn*hd]. This vector data is a random Gaussian noise vector, where the first dimension b represents the number of videos to be generated; the second dimension t is the temporal dimension, used to represent the number of frames in a video; the third dimension s is the spatial dimension, used to represent the number of pixels in each frame; and the fourth dimension hn*hd is used for subsequent attention calculations, where hn represents the number of heads in the attention mechanism and hd represents the size of the hidden layer. Of these four dimensions, only two need to be calculated: the temporal dimension t and the spatial dimension s.

[0077] In the above S401, data with multiple dimensions can be divided into N parts according to the first dimension. The i-th computing device can obtain the i-th part of the data from the N parts of the data as the i-th first vector data. That is, the i-th first vector data can be understood as the i-th part of the data after dividing the data with multiple dimensions into N parts according to the first dimension.

[0078] It should be understood that when data with multiple dimensions is split according to the first dimension, it can be split uniformly or non-uniformly; no limitation is made here. For ease of explanation, the following text will use the example of data with multiple dimensions being split uniformly.

[0079] It should be understood that the first dimension can be one of the X dimensions that need to be calculated. For example, in a video generation scenario, the first dimension can be a time dimension or a spatial dimension.

[0080] In this embodiment, the j-th computing device in the computing device cluster 202 also performs the steps in S401 described above, that is, the j-th computing device obtains the j-th second vector data processed by the j-th computing device from the vector data processed by the N computing devices, where j is an integer between [1, N] and j is not equal to i. The j-th second vector data can be understood as the j-th part of the data after dividing the data of multiple dimensions into N parts according to the first dimension.

[0081] For example, in a video generation scenario, the vector data is a 4-dimensional vector [b,t,s,hn*hd]. With the first dimension being the time dimension, the vector data is divided into N parts according to the time dimension, each part being a 4-dimensional vector [b,t / N,s,hn*hd]. The i-th first vector data is the i-th part of the N parts, hence the i-th first vector data is a 4-dimensional vector [b,t / N,s,hn*hd]. The j-th second vector data is the j-th part of the N parts, hence the j-th second vector data is a 4-dimensional vector [b,t / N,s,hn*hd].

[0082] S402, the i-th computing device segments the i-th first vector data according to the first dimension to obtain M first sub-vector data. Where M is an integer greater than 1.

[0083] In the above S402, after the i-th computing device obtains the preset segmentation component M, it can segment the i-th first vector data according to the first dimension based on the preset segmentation component M to obtain M first sub-vector data.

[0084] For example, in a video generation scenario, the i-th first vector data is a 4-dimensional vector [b,t / N,s,hn*hd]. According to the preset segmentation component M, the i-th computing device segments the i-th first vector data according to the time dimension to obtain M first sub-vector data. Each of the M first sub-vector data can be denoted as a 4-dimensional vector [b,t / N / M,s,hn*hd].

[0085] It should be understood that when the i-th first vector data is segmented according to the first dimension, it can be a uniform segmentation or a non-uniform segmentation; no limitation is made here. For ease of explanation, the following text will use the example of the i-th first vector data being uniformly segmented.

[0086] In this embodiment, the j-th computing device also performs the steps in S402 described above. For ease of description later, the j-th computing device segments the j-th second vector data according to the first dimension to obtain M second sub-vector data.

[0087] For example, in a video generation scenario, the j-th second vector data is a 4-dimensional vector [b,t / N,s,hn*hd]. According to the preset segmentation component M, the j-th computing device segments the j-th second vector data according to the time dimension to obtain M second sub-vector data. Each of the M second sub-vector data can be denoted as a 4-dimensional vector [b,t / N / M,s,hn*hd].

[0088] It should be understood that when the j-th second vector data is segmented according to the first dimension, it can be a uniform segmentation or a non-uniform segmentation; no limitation is made here. Furthermore, the segmentation method of the j-th second vector data is the same as the segmentation method of the i-th first vector data. For ease of explanation, the following text will use the uniform segmentation of the j-th second vector data as an example.

[0089] S403, after the i-th computing device completes the calculation of the second dimension for the m-th first sub-vector data, the i-th computing device sends the calculation result of the second dimension corresponding to the m-th first sub-vector data to the other N-1 computing devices, and receives the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices. Simultaneously, it performs the calculation of the second dimension for the (m+1)-th first sub-vector data, until the calculation result of the second dimension corresponding to the M-th first sub-vector data is sent to the other N-1 computing devices, and the i-th computing device receives the calculation result of the second dimension corresponding to the M-th second sub-vector data sent by each of the other N-1 computing devices. Here, m is an integer taken from [1, M-1].

[0090] In step S403 above, the i-th computing device can sequentially perform second-dimensional calculations on M first sub-vector data, and the calculation result of the second dimension corresponding to any first sub-vector data can be denoted as a 4-dimensional vector [b,t / N / M,s,hn*hd]. Simultaneously, the j-th computing device can sequentially perform second-dimensional calculations on M second sub-vector data, and the calculation result of the second dimension corresponding to any second sub-vector data can be denoted as a 4-dimensional vector [b,t / N / M,s,hn*hd].

[0091] In this embodiment, after the i-th computing device completes the calculation of the second dimension for the m-th first sub-vector data, the i-th computing device can send the calculation result of the second dimension corresponding to the m-th first sub-vector data to the other N-1 computing devices through parallel communication among the N computing devices, and receive the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices. The second dimension can be one of the X dimensions that need to be calculated, and the second dimension is different from the first dimension. For example, in a video generation scenario, if the first dimension is the time dimension, the second dimension is the spatial dimension.

[0092] In one possible implementation, parallel communication between N computing devices can be achieved through either implementation A1 or A2.

[0093] In implementation method A1, parallel communication between N computing devices can be achieved using the all-to-all operator in the ensemble communication library. This allows full utilization of the existing ensemble communication library without introducing additional communication operators to achieve parallel communication between N computing devices.

[0094] In implementation method A2, when the all-to-all operator in the collection communication library cannot be used, other communication operators can be introduced to achieve parallel communication between N computing devices.

[0095] In S403 above, the i-th computing device sends the calculation result of the second dimension corresponding to the m-th first sub-vector data to the other N-1 computing devices, and receives the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices. This can be achieved through methods such as... Figure 5 The following steps are performed as shown.

[0096] S501, the i-th computing device segments the calculation result of the second dimension corresponding to the m-th first sub-vector data in the second dimension according to the number N of computing devices in the computing device cluster, and obtains N calculation result fragments corresponding to the m-th first sub-vector data.

[0097] In the above S501, the i-th computing device divides the calculation result of the second dimension corresponding to the m-th first sub-vector data into N segments in the second dimension according to the number N computing devices in the computing device cluster, so as to obtain the N calculation result segments corresponding to the m-th first sub-vector data.

[0098] It should be understood that when the i-th computing device segments the second dimension corresponding to the m-th first sub-vector data, it can either uniformly segment the second dimension or use other segmentation methods, which are not limited here. For ease of explanation, uniform segmentation will be used as an example in the following text.

[0099] When the calculation result of the second dimension corresponding to the m-th first sub-vector data is a 4-dimensional vector [b,t / N / M,s,hn*hd], the i-th computing device segments the calculation result of the second dimension corresponding to the m-th first sub-vector data in the second dimension to obtain N calculation result fragments corresponding to the m-th first sub-vector data. Among them, each of the N calculation result fragments corresponding to the m-th first sub-vector data is a 4-dimensional vector [b,t / N / M,s / M,hn*hd].

[0100] During the execution of S501 by the i-th computing device, the j-th computing device performs the following steps: The j-th computing device, based on the number N of computing devices in the computing device cluster, segments the calculation result of the second dimension corresponding to the m-th second sub-vector data in the second dimension to obtain N calculation result fragments corresponding to the m-th second sub-vector data.

[0101] It should be understood that when the j-th computing device segments the second dimension corresponding to the m-th second sub-vector data, it can either uniformly segment the second dimension or use other segmentation methods, which are not limited here. The j-th computing device uses the same segmentation method as the i-th computing device.

[0102] When the calculation result of the second dimension corresponding to the m-th second sub-vector data is a 4-dimensional vector [b,t / N / M,s,hn*hd], the j-th computing device segments the calculation result of the second dimension corresponding to the m-th second sub-vector data in the second dimension to obtain N calculation result fragments corresponding to the m-th second sub-vector data. Among them, each of the N calculation result fragments corresponding to the m-th second sub-vector data is a 4-dimensional vector [b,t / N / M,s / M,hn*hd].

[0103] S502, the i-th computing device sends the j-th calculation result fragment from the N calculation result fragments corresponding to the m-th first sub-vector data to the j-th computing device in the computing device cluster, and receives the i-th calculation result fragment from the N calculation result fragments corresponding to the m-th second sub-vector data sent by the j-th computing device.

[0104] During the execution of S502 by the i-th computing device, the j-th computing device performs the following steps: the j-th computing device sends the i-th calculation result fragment from the N calculation result fragments corresponding to the m-th second sub-vector data to the i-th computing device, and receives the j-th calculation result fragment from the N calculation result fragments corresponding to the m-th first sub-vector data sent by the i-th computing device.

[0105] In this embodiment of the application, the i-th computing device can acquire M computation result fragments processed by each computing device, for a total of N*M computation result fragments.

[0106] Figure 5 The steps shown provide a specific method for the i-th computing device to send the calculation result of the second dimension corresponding to the m-th first sub-vector data, and to receive the calculation result of the second dimension corresponding to the m-th second sub-vector data.

[0107] S404, the i-th computing device generates the i-th vector result based on the calculation results corresponding to the M first sub-vector data respectively, and the calculation results corresponding to the M second sub-data vectors sent by each of the other N-1 computing devices.

[0108] In one possible implementation, the i-th computing device can generate the i-th vector result based on the i-th calculation result fragment corresponding to each of the M first sub-vector data and the i-th calculation result fragment corresponding to each of the M second sub-vector data sent by the j-th computing device.

[0109] In one possible implementation, the i-th computing device generates the i-th vector result based on the i-th calculation result fragment corresponding to each of the M first sub-vector data and the i-th calculation result fragment corresponding to each of the M second sub-vector data sent by the j-th computing device. This can be achieved through methods such as... Figure 6 The following steps are performed as shown.

[0110] S601, the i-th computing device sorts the i-th calculation result fragments corresponding to the M first sub-vector data and the i-th calculation result fragments corresponding to the M second sub-vector data sent by the j-th computing device according to the numbers of the N computing devices, and obtains the first sorting result.

[0111] In step S601 above, the i-th computing device sorts the i-th computation result fragments corresponding to the M first sub-vector data and the i-th computation result fragments corresponding to the M second sub-vector data sent by the j-th computing device in ascending order according to the numbers of the N computing devices, to obtain a first sorting result. The first sorting result includes N*M computation result fragments. In the first sorting result, the M i-th computation result fragments corresponding to the computing device with the smaller number precede the M i-th computation result fragments corresponding to the computing device with the larger number.

[0112] S602, the i-th computing device concatenates the first sorting results along the first dimension to obtain the i-th vector result.

[0113] In step S602 above, the first sorting result includes N*M calculation result segments, each of which is a 4-dimensional vector [b, t / N / M, s / M, hn*hd]. The i-th computing device concatenates the N*M calculation result segments from the first sorting result in the first dimension to obtain the i-th vector result, which is also a 4-dimensional vector [b, t, s / M, hn*hd]. The i-th vector result is the calculation result of the i-th first vector data in the second dimension.

[0114] Figure 6 The steps shown provide a specific method for generating the i-th vector result.

[0115] For example, such as Figure 7 As shown, the computing device cluster includes four computing devices, namely the first computing device, the second computing device, the third computing device, and the fourth computing device. In this embodiment, the i-th computing device is defined as the first computing device in the computing device cluster, and the other N-1 computing devices are defined as the second computing device, the third computing device, and the fourth computing device in the computing device cluster.

[0116] In the video generation scenario, the computing device cluster processes a 4-dimensional vector [2, 64, 1200, 1152]. The first dimension, 2, represents the number of videos to be generated; the second dimension, 64, is the temporal dimension, representing the number of frames in a video; the third dimension, 1200, is the spatial dimension, representing the number of pixels in each frame; and the fourth dimension, 1152, is used for subsequent attention calculations. The first dimension is the temporal dimension, and the second dimension is the spatial dimension.

[0117] The vector data [2,64,1200,1152] is divided into 4 parts according to the first dimension. Each part is a 4-dimensional vector [2,16,1200,1152]. The first computing device obtains the first part of the data as the first first vector data [2,16,1200,1152]. The second computing device obtains the second part of the data as the second second vector data [2,16,1200,1152]. The third computing device obtains the third part of the data as the third second vector data [2,16,1200,1152]. The fourth computing device obtains the fourth part of the data as the fourth second vector data [2,16,1200,1152].

[0118] For the first computing device, the first computing device divides the first first vector data [2,16,1200,1152] according to the time dimension based on the preset segmentation component 4, and obtains 4 first sub-vector data, which are respectively denoted as the first first sub-vector data, the second first sub-vector data, the third first sub-vector data, and the fourth first sub-vector data. Among them, the above 4 first sub-vector data are all 4-dimensional vectors [2,4,1200,1152].

[0119] For the second computing device, the second computing device divides the second second vector data [2,16,1200,1152] according to the time dimension based on the preset segmentation component 4, and obtains 4 second sub-vector data, which are respectively denoted as the first second sub-vector data, the second second sub-vector data, the third second sub-vector data, and the fourth second sub-vector data corresponding to the second computing device. Among them, the above 4 second sub-vector data are all 4-dimensional vectors [2,4,1200,1152].

[0120] For the third computing device, the third computing device divides the third second vector data [2,16,1200,1152] according to the time dimension based on the preset segmentation component 4, and obtains 4 second sub-vector data, which are respectively denoted as the first second sub-vector data, the second second sub-vector data, the third second sub-vector data, and the fourth second sub-vector data corresponding to the third computing device. Among them, the above 4 second sub-vector data are all 4-dimensional vectors [2,4,1200,1152].

[0121] For the fourth computing device, the fourth computing device divides the fourth second vector data [2,16,1200,1152] according to the time dimension based on the preset segmentation component 4, and obtains four second sub-vector data, which are respectively denoted as the first second sub-vector data, the second second sub-vector data, the third second sub-vector data, and the fourth second sub-vector data corresponding to the fourth computing device. Among them, the above four second sub-vector data are all 4-dimensional vectors [2,4,1200,1152].

[0122] For the first sub-vector data [2,4,1200,1152] in the first computing device, after the first computing device performs the spatial dimension calculation on the first sub-vector data, according to the number of computing devices in the computing device cluster (4), the spatial dimension calculation result corresponding to the first sub-vector data is spatially segmented to obtain four calculation result fragments corresponding to the first sub-vector data, which are respectively denoted as the first calculation result fragment, the second calculation result fragment, the third calculation result fragment, and the fourth calculation result fragment corresponding to the first sub-vector data. All four calculation result fragments corresponding to the first sub-vector data are [2,4,300,1152]. Subsequently, the first computing device retains the first calculation result fragment [2,4,300,1152] corresponding to the first first sub-vector data, and uses parallel communication between the four computing devices to simultaneously send the second calculation result fragment [2,4,300,1152] corresponding to the first first sub-vector data to the second computing device, the third calculation result fragment [2,4,300,1152] corresponding to the first first sub-vector data to the third computing device, and the fourth calculation result fragment [2,4,300,1152] corresponding to the first first sub-vector data to the fourth computing device.

[0123] Meanwhile, the processing of the first second sub-vector data in the second computing device, the first second sub-vector data in the third computing device, and the first second sub-vector data in the fourth computing device all refer to the processing process of the first first sub-vector data in the first computing device, which will not be repeated here.

[0124] Meanwhile, during the process of the first computing device sending the calculation result fragment corresponding to the first first sub-vector data to other computing devices, the first computing device can also receive the first calculation result fragment [2,4,300,1152] among the four calculation result fragments corresponding to the first second sub-vector data in the second computing device sent by the second computing device; it can also receive the first calculation result fragment [2,4,300,1152] among the four calculation result fragments corresponding to the first second sub-vector data in the third computing device sent by the third computing device; and it can also receive the first calculation result fragment [2,4,300,1152] among the four calculation result fragments corresponding to the first second sub-vector data in the fourth computing device sent by the fourth computing device.

[0125] Simultaneously, while the first computing device is sending the calculation result fragment corresponding to the first first sub-vector data to other computing devices, the first computing device can also perform spatial dimension calculations on the second first sub-vector data. Specifically, the calculation of the spatial dimension by the first computing device on the second first sub-vector data can be performed with reference to the calculation of the spatial dimension by the first computing device on the first first sub-vector data.

[0126] The processing method for the second, third, and fourth first sub-vector data in the first computing device is the same as that for the first first sub-vector data in the first computing device, and will not be repeated here.

[0127] The processing method for the second, third, and fourth sub-vector data in the second computing device is the same as that for the first sub-vector data in the first computing device, and will not be repeated here.

[0128] The processing method for the second, third, and fourth sub-vector data in the third computing device is the same as that for the first sub-vector data in the first computing device, and will not be repeated here.

[0129] The processing method for the second, third, and fourth sub-vector data in the fourth computing device is the same as that for the first sub-vector data in the first computing device, and will not be repeated here.

[0130] Finally, the first computing device can obtain 4*4 computation result fragments, each of which is [2,4,300,1152]. These 4*4 computation result fragments include the 4 computation result fragments generated by the first computing device, the 4 computation result fragments generated by the second computing device, the 4 computation result fragments generated by the third computing device, and the 4 computation result fragments generated by the fourth computing device.

[0131] The first computing device sorts the 16 (16 = 4 * 4) computation result fragments in ascending order according to the numbers of the four computing devices to obtain the first sorting result. The first sorting result is then concatenated in the time dimension to obtain the first vector result. The first vector result is [2, 4 * 16, 300, 1152].

[0132] The second computing device can obtain the second vector result by referring to the above method. The second vector result is [2,4*16,300,1152].

[0133] The third computing device can obtain the third vector result by referring to the above method. The third vector result is [2,4*16,300,1152].

[0134] The fourth computing device can obtain the fourth vector result by referring to the above method. The fourth vector result is [2, 4*16, 300, 1152].

[0135] S405, if the second dimension is not the last dimension to be calculated, then the i-th computing device splits the generated i-th vector result in the second dimension so that the split data can be calculated in the third dimension; if the second dimension is the last dimension to be calculated, then the i-th computing device uses the i-th vector result as the i-th calculation result of the vector data in the X dimensions.

[0136] In step S405 above, if the second dimension is not the last dimension to be calculated, the i-th computing device segments the generated i-th vector result along the second dimension, and then calculates the segmented data along the third dimension. This process is repeated until all X dimensions have been calculated. Here, the first, second, and third dimensions are one of the X dimensions to be calculated, and all three dimensions are distinct.

[0137] In S405 above, if the second dimension is the last dimension to be calculated, then the i-th computing device will use the i-th vector result as the i-th calculation result of the vector data in the X dimensions. The calculation results corresponding to the N computing devices are the overall calculation result of the vector data in the X dimensions.

[0138] Figure 4In the steps shown, the i-th computing device does not perform the calculation of the second dimension on the i-th first vector data and then send the calculation result of the second dimension corresponding to the i-th first vector data to other computing devices. Instead, it first divides the i-th first vector data according to the first dimension to obtain M first sub-vector data. After performing the calculation of the second dimension on the m-th first sub-vector data among the M first sub-vector data, it sends the calculation result of the second dimension corresponding to the m-th first sub-vector data to the other N-1 computing devices, and receives the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices. At the same time, it performs the calculation of the second dimension on the (m+1)-th first sub-vector data. As can be seen from the above process, the i-th computing device simultaneously executes the calculation result of the second dimension corresponding to the m-th first sub-vector data, receives the calculation result of the second dimension corresponding to the m-th second sub-vector data from each of the other N-1 computing devices, and performs the calculation of the second dimension on the m+1-th first sub-vector data. The above process can save data processing time in model training or inference, thereby reducing the duration of model training or inference.

[0139] Based on the above embodiments, this application also provides a data processing apparatus, which is used in... Figure 2 The i-th computing device in the computing device cluster shown is used to implement Figure 4 The illustrated embodiment provides a data processing method. See also... Figure 8 As shown, the data processing device includes: an acquisition module 801, a segmentation module 802, a synchronous execution module 803, and a generation module 804. Wherein:

[0140] The acquisition module 801 is used to acquire the i-th first vector data processed by the i-th computing device from the vector data processed by N computing devices. The vector data includes data of multiple dimensions. The i-th first vector data is the i-th part of the data after the data of multiple dimensions is divided into N parts according to the first dimension; i is any integer between [1, N] and N is an integer greater than 1.

[0141] The segmentation module 802 is used to segment the i-th first vector data according to the first dimension to obtain M first sub-vector data; M is an integer greater than 1;

[0142] The synchronous execution module 803 is used to, after performing the calculation of the second dimension on the m-th first sub-vector data, send the calculation result of the second dimension corresponding to the m-th first sub-vector data to the other N-1 computing devices, and receive the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices. At the same time, it performs the calculation of the second dimension on the (m+1)-th first sub-vector data until the calculation result of the second dimension corresponding to the M-th first sub-vector data is sent to the other N-1 computing devices, and receives the calculation result of the second dimension corresponding to the M-th second sub-vector data sent by each of the other N-1 computing devices, where m is an integer taken from [1, M-1].

[0143] The generation module 804 is used to generate the i-th vector result based on the calculation results of the second dimension corresponding to the M first sub-vector data respectively, and the calculation results of the second dimension corresponding to the M second sub-data vectors sent by each of the other N-1 computing devices. The i-th vector result is the calculation result of the i-th first vector data in the second dimension.

[0144] In one possible design, the synchronous execution module 803 is further configured to: if there are X dimensions that need to be calculated among multiple dimensions, and if the second dimension is not the last dimension that needs to be calculated, then the generated i-th vector result is split in the second dimension so that the split data can be calculated in the third dimension; if the second dimension is the last dimension that needs to be calculated, then the i-th vector result is used as the i-th calculation result of the vector data in the X dimensions, where X is an integer greater than 1.

[0145] In one possible design, the synchronous execution module 803 is specifically used to: send the calculation result corresponding to the m-th first sub-vector data to the other N-1 computing devices through parallel communication between N computing devices, and receive the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices.

[0146] In one possible design, the synchronous execution module 803 is specifically used to: send the j-th calculation result fragment of the N calculation result fragments corresponding to the m-th first sub-vector data to the j-th computing device in the computing device cluster, and receive the i-th calculation result fragment of the N calculation result fragments corresponding to the m-th second sub-vector data sent by the j-th computing device; j is an integer between [1, N], and j is not equal to i; the N calculation result fragments corresponding to the m-th first sub-vector data are obtained by the i-th computing device by splitting the calculation result of the second dimension corresponding to the m-th first sub-vector data in the second dimension according to the number N of computing devices in the computing device cluster; the N calculation result fragments corresponding to the m-th second sub-vector data are obtained by the j-th computing device by splitting the calculation result of the second dimension corresponding to the m-th second sub-vector data in the second dimension according to the number N of computing devices in the computing device cluster.

[0147] The generation module is specifically used to generate the i-th vector result based on the i-th calculation result fragment corresponding to each of the M first sub-vector data and the i-th calculation result fragment corresponding to each of the M second sub-vector data sent by the j-th computing device.

[0148] In one possible design, the generation module 804 is specifically used to: sort the i-th calculation result fragments corresponding to the M first sub-vector data and the i-th calculation result fragments corresponding to the M second sub-vector data sent by the j-th computing device according to the numbers of the N computing devices, and obtain a first sorting result; and concatenate the first sorting results in the first dimension to obtain the i-th vector result.

[0149] In one possible design, the vector data is the data corresponding to the multimodal model performing training or inference tasks, the calculation is an attention calculation, and the calculation of the X dimensions of data does not affect each other.

[0150] In this embodiment, the acquisition module, segmentation module, synchronous execution module, and generation module can all be implemented in software or in hardware. For example, the implementation of the synchronous execution module will be described below. Similarly, the implementation of the acquisition module, segmentation module, and generation module can refer to the implementation of the synchronous execution module.

[0151] As an example of a software functional unit, a synchronous execution module may include code running on a compute instance. A compute instance may include at least one of a physical host (computing device), a virtual machine, or a container. Furthermore, the aforementioned compute instance may be one or more. For example, a synchronous execution module may include code running on multiple hosts / virtual machines / containers. It should be noted that the multiple hosts / virtual machines / containers used to run the code may be distributed within the same region or in different regions. Further, the multiple hosts / virtual machines / containers used to run the code may be distributed within the same availability zone (AZ) or in different AZs, each AZ comprising one or more geographically proximate data centers. Typically, a region may include multiple AZs.

[0152] Similarly, multiple hosts / virtual machines / containers used to run this code can be distributed within the same Virtual Private Cloud (VPC) or across multiple VPCs. Typically, a VPC is set up within a region. Communication between two VPCs within the same region, as well as between VPCs in different regions, requires a communication gateway to be set up within each VPC to enable interconnection between VPCs.

[0153] As an example of a hardware functional unit, a synchronous execution module may include at least one computing device, such as a server. Alternatively, a synchronous execution module may also be a device implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD). The aforementioned PLD may be implemented using a complex programmable logical device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof.

[0154] The multiple computing devices included in the synchronous execution module can be distributed within the same region or in different regions. Similarly, the multiple computing devices included in the synchronous execution module can be distributed within the same Availability Zone (AZ) or in different AZs. Likewise, the multiple computing devices included in the synchronous execution module can be distributed within the same Virtual Private Cloud (VPC) or multiple VPCs. These multiple computing devices can be any combination of computing devices such as servers, ASICs, PLDs, CPLDs, FPGAs, and GALs.

[0155] It should be noted that, in other embodiments, the acquisition module is used to acquire the i-th first vector data processed by the i-th computing device from the vector data processed by N computing devices. The vector data includes data of multiple dimensions, and the i-th first vector data is the i-th part of the data after dividing the data of multiple dimensions into N parts according to the first dimension; i is any integer between [1, N], and N is an integer greater than 1; the segmentation module is used to segment the i-th first vector data according to the first dimension to obtain M first sub-vector data; M is an integer greater than 1; the synchronous execution module is used to send the calculation result of the second dimension corresponding to the m-th first sub-vector data to the other N-1 computing devices after performing the calculation of the second dimension on the m-th first sub-vector data, and receive the calculation result of the second dimension from each of the other N-1 computing devices. The calculation result of the second dimension corresponding to the m-th second sub-vector data is sent, and the calculation of the second dimension of the (m+1)-th first sub-vector data is performed simultaneously until the calculation result of the second dimension corresponding to the M-th first sub-vector data is sent to the other N-1 computing devices, and the calculation result of the second dimension corresponding to the M-th second sub-vector data sent by each of the other N-1 computing devices is received, where m is an integer between [1, M-1]. The generation module is used to generate the i-th vector result based on the calculation results of the second dimension corresponding to the M first sub-vector data and the calculation results of the second dimension corresponding to the M second sub-vector data sent by each of the other N-1 computing devices. The i-th vector result is the calculation result of the second dimension of the i-th first vector data.

[0156] The steps implemented by the acquisition module, segmentation module, synchronous execution module, and generation module can be specified as needed. By implementing different steps in the data processing method through the acquisition module, segmentation module, synchronous execution module, and generation module, the full functionality of the data processing device can be achieved.

[0157] This application also provides a computing device, which includes a processor and a memory, wherein the memory stores computer-executable instructions; the processor is used to execute the computer-executable instructions stored in the memory, and the processor can execute data processing methods.

[0158] This application also provides a computer program product containing instructions. The computer program product may be a software or program product containing instructions, capable of running on a computing device or stored on any usable medium. When the computer program product is run on at least one computing device, it causes the at least one computing device to perform a data processing method.

[0159] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center that includes one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct a computing system to perform a data processing method.

[0160] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of the present invention.

[0161] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0163] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0165] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.

Claims

1. A data processing method, characterized in that, The method, applied to the i-th computing device out of N computing devices, where i is any integer between [1, N] and N is an integer greater than 1, includes: The first vector data processed by the i-th computing device is obtained from the vector data processed by the N computing devices. The vector data includes data of multiple dimensions, and the first vector data is the i-th part of the data after the multiple dimensions of the data are divided into N parts according to the first dimension. The i-th first vector data is divided according to the first dimension to obtain M first sub-vector data; M is an integer greater than 1; After performing the calculation of the second dimension on the m-th first sub-vector data, the calculation result of the second dimension corresponding to the m-th first sub-vector data is sent to the other N-1 computing devices, and the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices is received. At the same time, the calculation of the second dimension is performed on the (m+1)-th first sub-vector data, until the calculation result of the second dimension corresponding to the M-th first sub-vector data is sent to the other N-1 computing devices, and the calculation result of the second dimension corresponding to the M-th second sub-vector data sent by each of the other N-1 computing devices is received, where m is an integer taken from [1, M-1]. Based on the calculation results of the second dimension corresponding to the M first sub-vector data respectively, and the calculation results of the second dimension corresponding to the M second sub-data vectors sent by each of the other N-1 computing devices, the i-th vector result is generated, and the i-th vector result is the calculation result of the i-th first vector data in the second dimension.

2. The method as described in claim 1, characterized in that, The method further includes: if there are X dimensions that need to be calculated among the multiple dimensions, and if the second dimension is not the last dimension that needs to be calculated, then the generated i-th vector result is split in the second dimension so that the split data is calculated in the third dimension; if the second dimension is the last dimension that needs to be calculated, then the i-th vector result is taken as the i-th calculation result of the vector data in the X dimensions, where X is an integer greater than 1.

3. The method as described in claim 1, characterized in that, The step of sending the calculation result corresponding to the m-th first sub-vector data to the other N-1 computing devices, and receiving the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices, includes: Through parallel communication among the N computing devices, the calculation result corresponding to the m-th first sub-vector data is sent to the other N-1 computing devices, and the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices is received.

4. The method according to any one of claims 1-3, characterized in that, The step of sending the calculation result of the second dimension corresponding to the m-th first sub-vector data to the other N-1 computing devices, and receiving the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices, includes: The system sends the j-th calculation result fragment from the N calculation result fragments corresponding to the m-th first sub-vector data to the j-th computing device in the computing device cluster, and receives the i-th calculation result fragment from the N calculation result fragments corresponding to the m-th second sub-vector data sent by the j-th computing device; j is an integer between [1, N], and j is not equal to i; the N calculation result fragments corresponding to the m-th first sub-vector data are obtained by the i-th computing device by splitting the calculation result of the second dimension corresponding to the m-th first sub-vector data in the second dimension according to the number N of computing devices in the computing device cluster; the N calculation result fragments corresponding to the m-th second sub-vector data are obtained by the j-th computing device by splitting the calculation result of the second dimension corresponding to the m-th second sub-vector data in the second dimension according to the number N of computing devices in the computing device cluster. The step of generating the i-th vector result based on the calculation results of the second dimension corresponding to the M first sub-vector data, and the calculation results of the second dimension corresponding to the M second sub-data vectors sent by each of the other N-1 computing devices, includes: The i-th vector result is generated based on the i-th calculation result fragment corresponding to each of the M first sub-vector data and the i-th calculation result fragment corresponding to each of the M second sub-vector data sent by the j-th computing device.

5. The method as described in claim 4, characterized in that, The step of generating the i-th vector result based on the i-th calculation result fragment corresponding to each of the M first sub-vector data and the i-th calculation result fragment corresponding to each of the M second sub-vector data sent by the j-th computing device includes: Based on the numbers of the N computing devices, the i-th calculation result fragments corresponding to the M first sub-vector data and the i-th calculation result fragments corresponding to the M second sub-vector data sent by the j-th computing device are sorted to obtain a first sorting result; The first sorting results are concatenated along the first dimension to obtain the i-th vector result.

6. The method according to any one of claims 1-5, characterized in that, The vector data is the data corresponding to the multimodal model performing training or inference tasks, the calculation is attention calculation, and the calculation of the X dimensions of data does not affect each other.

7. A data processing apparatus, characterized in that, include: The acquisition module is used to acquire the i-th first vector data processed by the i-th computing device from the vector data processed by N computing devices. The vector data includes data of multiple dimensions, and the i-th first vector data is the i-th part of the data after the data of multiple dimensions is divided into N parts according to the first dimension. i is any integer between [1, N], and N is an integer greater than 1; The segmentation module is used to segment the i-th first vector data according to the first dimension to obtain M first sub-vector data; M is an integer greater than 1; The synchronous execution module is used to, after performing the calculation of the second dimension on the m-th first sub-vector data, send the calculation result of the second dimension corresponding to the m-th first sub-vector data to the other N-1 computing devices, and receive the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices. At the same time, it performs the calculation of the second dimension on the (m+1)-th first sub-vector data until the calculation result of the second dimension corresponding to the M-th first sub-vector data is sent to the other N-1 computing devices, and receives the calculation result of the second dimension corresponding to the M-th second sub-vector data sent by each of the other N-1 computing devices, where m is an integer taken from [1, M-1]. The generation module is used to generate the i-th vector result based on the calculation results of the second dimension corresponding to the M first sub-vector data respectively, and the calculation results of the second dimension corresponding to the M second sub-data vectors sent by each of the other N-1 computing devices, wherein the i-th vector result is the calculation result of the i-th first vector data in the second dimension.

8. The apparatus as claimed in claim 7, characterized in that, The synchronous execution module is further configured to: if there are X dimensions that need to be calculated among the multiple dimensions, and if the second dimension is not the last dimension that needs to be calculated, then the generated i-th vector result is split in the second dimension so that the split data is calculated in the third dimension; if the second dimension is the last dimension that needs to be calculated, then the i-th vector result is used as the i-th calculation result of the vector data in the X dimensions, where X is an integer greater than 1.

9. The apparatus as claimed in claim 7, characterized in that, The synchronous execution module is specifically used for: Through parallel communication among the N computing devices, the calculation result corresponding to the m-th first sub-vector data is sent to the other N-1 computing devices, and the calculation result of the second dimension corresponding to the m-th second sub-vector data sent by each of the other N-1 computing devices is received.

10. The apparatus according to any one of claims 7-9, characterized in that, The synchronous execution module is specifically used for: Send the j-th calculation result fragment of the N calculation result fragments corresponding to the m-th first sub-vector data to the j-th computing device in the computing device cluster, and receive the i-th calculation result fragment of the N calculation result fragments corresponding to the m-th second sub-vector data sent by the j-th computing device; j is an integer that takes all elements between [1, N] and j is not equal to i; the N calculation result segments corresponding to the m-th first sub-vector data are obtained by the i-th computing device by splitting the calculation result of the second dimension corresponding to the m-th first sub-vector data in the second dimension according to the number N of computing devices in the computing device cluster; The N calculation result fragments corresponding to the m-th second sub-vector data are obtained by the j-th computing device by splitting the calculation result of the second dimension corresponding to the m-th second sub-vector data in the second dimension according to the number N of computing devices in the computing device cluster. The generation module is specifically used for: The i-th vector result is generated based on the i-th calculation result fragment corresponding to each of the M first sub-vector data and the i-th calculation result fragment corresponding to each of the M second sub-vector data sent by the j-th computing device.

11. The apparatus as claimed in claim 10, characterized in that, The generation module is specifically used for: Based on the numbers of the N computing devices, the i-th calculation result fragments corresponding to the M first sub-vector data and the i-th calculation result fragments corresponding to the M second sub-vector data sent by the j-th computing device are sorted to obtain a first sorting result; The first sorting results are concatenated along the first dimension to obtain the i-th vector result.

12. The apparatus according to any one of claims 7-11, characterized in that, The vector data is the data corresponding to the multimodal model performing training or inference tasks, the calculation is attention calculation, and the calculation of the X dimensions of data does not affect each other.

13. A computer program product containing instructions, characterized in that, When the instruction is executed by the computing device, the computing device performs the method as described in any one of claims 1-6.

14. A computer-readable storage medium, characterized in that, It includes computer program instructions, which, when executed by a computing device, cause the computing device to perform the method as described in any one of claims 1-6.

Images