Multi-modal data processing method and system, and related device

Abstract

Provided in the present application are a multi-modal data processing method and system, and a related device. The method is applied to a computing system comprising a management module and a plurality of computing modules. The method comprises: a management module acquiring multi-modal data, and then generating a plurality of data processing tasks on the basis of the multi-modal data, wherein the data processing tasks each comprise data of one modality in the multi-modal data and an operator corresponding to the modality, and the corresponding operator is used for encoding the data of the modality; the management module then issuing the plurality of data processing tasks to a plurality of computing modules; and the plurality of computing modules executing in parallel the respective received data processing tasks, so as to obtain feature representations corresponding to data of different modalities, wherein the feature representations corresponding to the data of different modalities are used for the training or inference of a multi-modal model. Data of different modalities in multi-modal data is processed in parallel by means of a plurality of computing modules, so that the efficiency of a multi-modal model can be improved.

Description

A multi-modal data processing method, system and related device

[0001] The present application claims priority from the Chinese patent application No. 202411370791.7 filed on September 27, 2024, and entitled "A multi-modal data processing method, system and related device", the content of which is incorporated herein by reference in its entirety. TECHNICAL FIELD

[0002] The present application relates to the field of computer technology, and in particular to a multi-modal data processing method, system and related device. BACKGROUND

[0003] A single-modal model refers to an artificial intelligence (AI) model that is trained and inferred using only one type of data. For example, a natural language processing model for text processing, a convolutional neural model for image classification and recognition, and a model for speech recognition, etc. With the development of technology, single-modal models have been unable to meet the needs of the current information explosion era, and multi-modal models have emerged. A multi-modal model refers to a deep learning model that can process and understand multiple modal data. The multi-modal model takes at least two modal data such as text, image, audio or video as input, extracts features from the input multi-modal data and fuses them, so that the multi-modal model can achieve more comprehensive and accurate understanding and inference capabilities. The input of the multi-modal model includes multiple types of data. As the diversity and complexity of data increase, the types of data that the multi-modal model needs to process will also increase. Therefore, how to improve the processing efficiency of multi-modal data is a technical problem that needs to be solved in the application of multi-modal models. SUMMARY

[0004] The present application provides a multi-modal data processing method, system and related device. The method can process different modal data in multi-modal data in parallel, and by processing different modal data in multiple multi-modal data in parallel, the efficiency of the multi-modal model can be improved.

[0005] In a first aspect, the present application provides a multi-modal data processing method for a computing system including a management module and a plurality of computing modules, the method comprising: obtaining, by the management module, multi-modal data; the multi-modal data including at least two of text data, image data, audio data, or video data; then generating, by the management module, a plurality of data processing tasks according to the multi-modal data, wherein each data processing task includes data of one modality in the multi-modal data and an operator corresponding to the one modality, and the operator in a data processing task is used to encode the data of the one modality included in the data processing task; after generating the plurality of data processing tasks, the management module distributes the plurality of data processing tasks to the plurality of computing modules, and each computing module is used to execute one data processing task in the plurality of data processing tasks; after receiving the respective data processing tasks, the plurality of computing modules execute the respective received data processing tasks in parallel to obtain feature representations corresponding to the data of different modalities; and the feature representations corresponding to the data of different modalities are used for training or inference of a multi-modal model.

[0006] After receiving a processing request of the multi-modal data, the management module generates a plurality of data processing tasks according to the multi-modal data, and then distributes the plurality of data processing tasks to the plurality of computing modules, so that the plurality of computing modules execute the respective received data processing tasks in parallel to encode the data of different modalities, which can improve the encoding efficiency of the multi-modal data and provide the efficiency of processing the multi-modal data by the multi-modal model.

[0007] In a possible implementation, each computing module in the plurality of computing modules includes at least one thread, or each computing module includes at least one process, or each computing module includes at least one core of an AI chip, or each computing module includes at least one AI chip, or each computing module includes at least one of any of the following computing resources: a server, etc. The management module can allocate computing resources for processing data of different modalities in the granularity of any of the above computing resources, reasonably utilize various computing resources, and improve the utilization rate of resources.

[0008] In a possible implementation, before distributing the plurality of data processing tasks to the plurality of computing modules, the management module further includes: determining, by the management module, the computing resources allocated to each computing module in the plurality of computing modules according to the data amount of the data of each modality in the multi-modal data, the computing resources being any of the following: a thread, a process, a core of an AI chip, an AI chip, or a server. The management module can perform load balancing according to the data amount of the data of each modality in the multi-modal data, so that the computing resources in the computing system can be reasonably utilized, for example, the data of a modality with a larger data amount is processed by more computing resources, the utilization rate of the computing resources in the computing system is improved, and the efficiency of the computing system is improved.

[0009] In a possible implementation, if the computing modules include multiple threads, multiple processes, multiple cores of AI chips, multiple AI chips, or multiple servers, the computing modules processing data of the same modality perform the received data processing tasks in a manner of data parallelism, model parallelism, tensor parallelism, pipeline parallelism, or hybrid parallelism, thereby improving the efficiency of data processing.

[0010] In a possible implementation, before the management module generates the multiple data processing tasks according to the multi-modal data, the management module performs preprocessing on the data of each modality in the multi-modal data in parallel. The management module performs preprocessing on the data of different modalities in parallel, thereby improving the efficiency of the management module in processing the multi-modal data.

[0011] In a possible implementation, the method further includes: performing a communication task between the multiple computing modules in the computing system, synchronizing the feature representations of the data of each modality obtained by respectively performing the corresponding data processing tasks; and then splicing, by each of the multiple computing modules, the feature representations of the data of each modality to obtain fusion features corresponding to the multi-modal data, wherein the fusion features corresponding to the multi-modal data are used for training or inference of a multi-modal model.

[0012] In a second aspect, the present application further provides a computing system, which includes a management module and multiple computing modules, the management module is configured to perform operations as described in the first aspect or any possible implementation of the first aspect, and the multiple computing modules are configured to perform operations as described in the first aspect or any possible implementation of the first aspect.

[0013] In a third aspect, the present application further provides a management module for a computing system including the management module and multiple computing modules, the computing system being configured to process multi-modal data; the management module includes a communication unit and a processing unit, wherein the communication unit is configured to obtain the multi-modal data; the multi-modal data includes at least two of text data, image data, audio data, or video data; the processing unit is configured to generate multiple data processing tasks according to the multi-modal data, wherein each data processing task includes data of one modality in the multi-modal data and an operator corresponding to the modality, and the operator in one data processing task is configured to encode the data of one modality included in the data processing task; the communication unit is further configured to distribute the multiple computing tasks to the multiple computing modules; so that the multiple computing modules perform the data processing tasks received respectively in parallel, to obtain feature representations of the data of different modalities; wherein each computing module is configured to perform one data processing task in the multiple data processing tasks, and the feature representations of the data of different modalities are used for training or inference of a multi-modal model.

[0014] In a possible implementation, the processing unit is further configured to determine, according to an amount of data of the data of each modality in the multi-modal data, an amount of computing resources allocated to each of the plurality of computing modules, the computing resources being threads, processes, cores of AI chips, AI chips, or servers, and each of the plurality of computing modules including at least one thread, at least one process, at least one core of an AI chip, at least one AI chip, or at least one server.

[0015] In a possible implementation, the processing unit is further configured to perform, in parallel, the preprocessing on the data of each modality in the multi-modal data.

[0016] In a fourth aspect, the present application provides a computing device, including a processor, at least one AI chip, and a memory, the processor executing instructions stored in the memory to implement operations performed by the management module in the first aspect or any possible implementation of the first aspect, and the at least one AI chip implementing operations performed by the plurality of computing modules in the first aspect or any possible implementation of the first aspect.

[0017] In a fifth aspect, the present application provides a computing device, including a processor and a memory, the processor executing instructions stored in the memory to implement operations performed by the management module in the second aspect or any possible implementation of the second aspect.

[0018] In a sixth aspect, the present application provides a computer-readable storage medium, including computer program instructions, when executed by a computing device, causing the computing device to perform operations performed by the management module in the second aspect or any possible implementation of the second aspect.

[0019] In a seventh aspect, the present application provides a computer program product including instructions, when executed by a computing device, causing the computing device to perform operations performed by the management module in the second aspect or any possible implementation of the second aspect.

[0020] On the basis of the implementation manners of the aspects described above, the present application can be further combined to provide more implementation manners. BRIEF DESCRIPTION OF DRAWINGS

[0021] FIG. 1 is a schematic diagram of a computing system provided by the present application;

[0022] FIG. 2 is a schematic diagram of a multi-modal data processing method provided by the present application;

[0023] FIG. 3 is a schematic diagram of processing multi-modal data provided by the present application;

[0024] FIG. 4 is a schematic diagram of a multi-modal data processing apparatus provided by the present application;

[0025] FIG. 5 is a schematic diagram of a computing device provided by the present application. DETAILED DESCRIPTION

[0026] Traditional single-modal models can only process one type of data such as text, image or speech, for example, processing images for image recognition, recognizing speech to realize speech-to-text, etc. However, problems in the real world often involve the combination of multiple modalities of data. For example, in a visual question answering task, the model needs to combine images and text descriptions to answer; in a medical diagnosis task, the model needs to combine medical images and doctors' instructions for diagnosis; in the tasks involved in intelligent driving, the model needs to combine video and radar data for decision-making, etc. In order to better understand and process complex problems, it is necessary to combine multiple modalities of data, so that the model can get more accurate and more comprehensive analysis.

[0027] A multi-modal model is an artificial intelligence (AI) model that can process and understand multi-modal data, which refers to data including two or more different modalities, including but not limited to text, image, audio, video and radar data, etc. Multi-modal data can provide rich information for multi-modal models. Through multiple different modalities of data, multi-modal models can more comprehensively understand the semantics and contextual information behind the data, and thus achieve more accurate understanding and reasoning capabilities.

[0028] However, the input of a multi-modal model includes multiple modalities of data, and as the diversity and complexity of data increase, the types of data that the multi-modal model needs to process will also increase, which will lead to an increase in the time it takes for the model to process data. Therefore, how to improve the processing efficiency of multi-modal data is a technical problem that needs to be solved in the application of multi-modal models.

[0029] The present application provides a multi-modal data processing method, which can parallel encode different modalities of data in the input multi-modal data to obtain feature representations corresponding to different modalities of data, and then concatenate the feature representations corresponding to different modalities of data to obtain data that can be input into a foundation model for inference. By parallel processing different modalities of data in multiple multi-modal data, the efficiency of the multi-modal model can be improved.

[0030] Referring to FIG. 1, FIG. 1 is a schematic diagram of a computing system provided by the present application, which can be used to implement the multi-modal data processing method provided by the present application. The computing system includes a management module and a plurality of computing modules. In the present application, one computing module includes a set of computing resources for processing data of one modality. For example, the computing resources for processing text data are the first computing module, the computing resources for processing image data are the second computing module, the computing resources for processing audio data are the third computing module, and the computing resources for processing video data are the fourth computing module. Each of the plurality of computing modules included in the computing system is used to process data of one modality. The computing resources can be any one of a thread, a process, a core in an AI chip, an AI chip, or a server. The management module can configure the computing resources included in each computing module at any granularity, such as a thread, a process, a core in an AI chip, an AI chip, or a server. Each server includes a plurality of AI chips. The AI chip can be a graphics processing unit (GPU), a neural network processing unit (NPU), a tensor processing unit (TPU), or other special-purpose AI chips.

[0031] The management module is used to obtain multi-modal data, and then distribute the multi-modal data to the plurality of computing modules. The plurality of computing modules perform parallel processing on the different modalities of data included in the multi-modal data. Each computing module encodes the data of one modality in the multi-modal data to obtain a feature representation corresponding to the data of each modality, and then concatenates the feature representations to obtain a fusion feature corresponding to the multi-modal data.

[0032] In the present application, the computing resources in the computing system can be scheduled at the granularity of a thread. Each of the plurality of computing modules includes one or more threads. When a computing module includes a plurality of threads, the plurality of threads can be threads on the same AI chip or threads on different AI chips.

[0033] The computing resources in the computing system can also be scheduled at the granularity of a process. Each of the plurality of computing modules includes one or more processes. When a computing module includes a plurality of processes, the plurality of processes can be processes on the same AI chip or processes on different AI chips.

[0034] The computing resources in the computing system can also be scheduled at the granularity of cores in an AI chip. Each of the plurality of computing modules includes one or more cores. When a computing module includes multiple cores, the multiple cores can be cores in the same AI chip or cores in different AI chips. For example, the cores in the same AI chip are used to process data of the same modality, or one or more cores in one AI chip and one or more cores in another AI chip are used to process data of the same modality together.

[0035] The computing system includes one or more AI chips, i.e., the plurality of cores included in the plurality of computing modules for processing data of different modalities can be cores in the same AI chip or cores in different AI chips. For example, the multi-modal data is processed in parallel by AI chips, and different cores process data of different modalities in parallel.

[0036] When the plurality of computing modules for processing multi-modal data are located in one AI chip, or the plurality of computing modules are distributed in multiple AI chips, and the multiple AI chips are located in the same computing device, the management module can be a central processing unit (CPU) in the computing device, or a thread or process in the computing device. When the plurality of AI chips included in the plurality of computing modules are distributed in multiple computing devices, the management module can be a CPU, a process or a thread in one of the computing devices, or another computing device other than the multiple computing devices.

[0037] The computing resources in the computing system can also be scheduled at the granularity of AI chips. Each of the plurality of computing modules includes one or more AI chips. When a computing module includes multiple AI chips, the multiple AI chips can be AI chips in the same server or AI chips in different servers. For example, the AI chips in the same server are used to process data of the same modality, or one or more AI chips in one server and one or more AI chips in another server are used to process data of the same modality together.

[0038] The computing system includes one or more servers, i.e., the plurality of AI chips included in the plurality of computing modules for processing data of different modalities can be AI chips in the same server or AI chips in different servers. For example, the multi-modal data is processed in parallel by AI chips in one server, and different AI chips are used to process data of different modalities.

[0039] The computing resources in the computing system can also be scheduled at the granularity of servers. Each of the plurality of computing modules includes one or more servers.

[0040] The processing method of multi-modal data provided in the present application will be described below with reference to the accompanying drawings. As shown in FIG. 2, FIG. 2 is a schematic diagram of a processing method of multi-modal data provided in the present application. The method includes S201-S205.

[0041] S201. The management module obtains multi-modal data.

[0042] The multi-modal data includes, but is not limited to, at least two of text data, audio data, image data, or video data. The multi-modal data can be data input by a user through a user terminal. After receiving the multi-modal data input by the user, the user terminal generates a data processing request, and uploads the data processing request to the management module. The data processing request includes the multi-modal data input by the user. For example, in an emotion analysis task, the user inputs a piece of text and a piece of video. The video contains speech. The text information can be "analyze the emotion that the video producer wants to express in combination with the video and the speech in the video". The multi-modal data can also be training data for training a multi-modal model. The management module can obtain the multi-modal data from a storage device according to a file path.

[0043] S202. The management module generates a plurality of data processing tasks according to the multi-modal data.

[0044] In the present application, the computing system includes a plurality of computing modules, each of which is used to process data of one modality. After receiving the multi-modal data, the management module generates a plurality of data processing tasks according to the multi-modal data. Each data processing task includes data of one modality in the multi-modal data and an operator corresponding to the data of the modality. The operator included in a data processing task is used to encode the data of one modality included in the data processing task.

[0045] In the present application, before generating the plurality of data processing tasks, the management module performs corresponding preprocessing on the data of different modalities, and then generates the data processing tasks according to the data after preprocessing. For example, the management module performs preprocessing operations such as word segmentation, stop word removal, stem extraction, and standardization on text data; performs preprocessing operations such as noise removal, frame division, and mel spectrum extraction on audio data; performs preprocessing operations such as image size adjustment, noise removal, and normalization on image data; and performs preprocessing operations such as frame extraction, image size adjustment, noise removal, and normalization on video data. If the video contains speech or subtitles, the speech in the video is extracted as audio data, and the subtitles in the video are extracted as text data. The management module performs the preprocessing operations on the data of different modalities in parallel through the computing resources included in the management module. For example, the management module is a CPU, and the management module performs the preprocessing operations on the data of each modality in the multi-modal data in parallel through a plurality of cores, or performs the preprocessing operations on the data of each modality in the multi-modal data in parallel through a plurality of threads or processes.

[0046] S203. The management module distributes the plurality of data processing tasks to the plurality of computing modules.

[0047] After generating the plurality of data processing tasks, the management module distributes the plurality of data processing tasks to the plurality of computing modules in the computing system, each computing module in the plurality of computing modules being configured to process one data processing task in the plurality of data processing tasks.

[0048] In a possible implementation, the computing modules for processing each modality in the computing system are pre-configured, and after generating the plurality of data processing tasks, the management module distributes each data processing task to a corresponding computing module according to the data type in each data processing task. For example, the computing resource is an NPU, the computing system includes a plurality of 10 NPUs, namely NPU0-NPU9, NPU0 is configured to process text data, NPU1 and NPU2 are configured to process audio data, NPU3-NPU5 are configured to process image data, and NPU6-NPU9 are configured to process video data. The first computing module for processing text data includes NPU0, the second computing module for processing audio data includes NPU1 and NPU2, the third computing module for processing image data includes NPU3-NPU5, and the fourth computing module for processing video data includes NPU6-NPU9.

[0049] If the data processing task includes text data, the management module distributes the data processing task including the text data to the first computing module; if the data processing task includes audio data, the management module distributes the data processing task including the audio data to the second computing module; if the data processing task includes image data, the management module distributes the data processing task including the image data to the third computing module; and if the data processing task includes video data, the management module distributes the data processing task including the video data to the fourth computing module.

[0050] In another possible implementation, after receiving the multi-modal data, the management module performs load balancing according to the data amount of different modal data in the multi-modal data and the total amount of computing resources in the computing system, and allocates different proportions of computing resources to process data of different modalities. For example, different proportions of computing resources are allocated to process data of different modalities according to the proportion of the storage space occupied by the data of different modalities, and the computing resources for processing data of one modality are proportional to the storage space occupied by the data of the modality; or for data of different modalities, the management module first determines the sequence length of the data of different modalities, and then allocates different proportions of computing resources to process data of different modalities according to the proportion between the sequence lengths corresponding to the data of different modalities.

[0051] For example, the multi-modal data includes text data, audio data and image data, and the ratio between the data amount of the text data, the audio data and the image data or the ratio between the sequence lengths is 1:2:3. The computing modules in the computing system are allocated according to the ratio, 1 / 6 of the computing resources is used to process the text data, and 1 / 2 of the computing resources is used to process the image data. Optionally, since the processing of the image data is more complex than the processing of the text data and the audio data, the application also sets a corresponding complexity coefficient for the data of different modalities. After determining the corresponding sequence lengths of the data of different modalities, the corresponding sequence length of each modality of data is multiplied by the corresponding complexity coefficient to obtain the final sequence length of the data of different modalities. Then, the computing resources for processing the data of different modalities are configured according to the ratio between the final sequence lengths. The complexity coefficient of the text data is less than the complexity coefficient of the audio data, the complexity coefficient of the audio data is less than the complexity coefficient of the image data, and the complexity coefficient of the image data is less than or equal to the complexity coefficient of the video data. For example, the complexity coefficient of the text data is 0.5, the complexity coefficient of the audio data is 1, and the complexity coefficients of the image data and the video data are both 3. It should be understood that the above examples are only used for illustration and cannot be understood as specific limitations. The management module can also allocate computing resources for processing data of different modalities by other strategies.

[0052] In determining the sequence length of the data of different modalities, for text data, the management module tokenizes the text data, and after pre-processing operations such as stop word removal, the remaining token data is the sequence length corresponding to the text data. For audio data, after pre-processing operations such as sampling and framing, the number of frames of the audio data is the sequence length corresponding to the audio data. For example, for 100 seconds of audio, if the sampling frequency is 16 kHz, the length of each frame is 20 milliseconds when framing, and the overlap rate between frames is 50%, then 100 seconds of audio data can be divided into 1600 frames, and the sequence length corresponding to the audio data is 1600. For image data, the image data includes one or more images, each image is divided into patches of the same size, and the sequence length corresponding to the image data is the sum of the number of patches obtained after dividing all images into patches. For example, the images in the image data are pre-processed, such as size adjustment and normalization, and the size of each image is adjusted to 224x224. For an image, if it is divided into 14x14 patches, 256 patches can be obtained, and the sequence length corresponding to the image is 256. If the multi-modal data includes 4 images, the sequence length corresponding to the image data is 1024. For video data, the video data is frame extracted to obtain multiple images, and the sequence length corresponding to the multiple images is taken as the sequence length of the video data. The method for determining the sequence length of the multiple images can refer to the method for determining the sequence length of the image data, which will not be described here.

[0053] It should be understood that after the computing resources are configured according to the ratio between the data amounts of the data according to different modalities or the ratio between the sequence lengths corresponding to the data of different modalities to process the data of different modalities, the computing modules corresponding to the data of different modalities can be determined. The management module assigns multiple data processing tasks to the corresponding computing modules respectively.

[0054] S204. Each computing module executes the data processing task received by itself in parallel, encodes the data included in the received data processing task, and obtains the feature representation corresponding to the data of different modalities.

[0055] After each computing module receives the corresponding data processing task, each computing module encodes the data in the received data processing task through an encoder and an operator to obtain the feature representation corresponding to the different modal data. Referring to FIG. 3, FIG. 3 is a schematic diagram of processing multi-modal data provided by the present application. After the first computing module receives a task including text data, the text data is encoded by a text encoder to convert the text data into corresponding text embedding, i.e., the feature representation corresponding to the text data. The text encoder can be an ELMo model or a BERT model, etc. After the second computing module receives a task including audio data, the audio data is encoded by an audio encoder to convert the audio data into corresponding audio embedding, i.e., the feature representation corresponding to the text data. The audio encoder can be a VGGish model, an OpenL3 model, a YAMNet model or a DeepSBD model, etc. After the third computing module receives a task including image data, the image data is encoded by an image encoder to convert the image data into corresponding image embedding, i.e., the feature representation corresponding to the image data. The image encoder can be a VGG model, a ResNet model, a ViT model or a CLIP model, etc. It should be understood that when processing video data, multiple frames of images are extracted from the video, and the multiple frames of images are processed to obtain video embedding corresponding to the video, so the processing of the video data can refer to the processing method of the image data described above, which will not be described here.

[0056] In the present application, the feature representation corresponding to the data of different modalities can be represented as (b_size, seq, dim), where b_size represents the number of data of one modality, seq represents the sequence length, and dim represents the dimension. For example, for a piece of text in the text data, if the token corresponding to the piece of text is 10, and the embedding corresponding to each token is 1024 dimensions, then the feature representation corresponding to the text data can be represented as (1, 10, 1024); if there are 6 images corresponding to the image data, and the sequence length corresponding to each image is 256, and the embedding corresponding to each patch is 2048 dimensions, then the feature representation corresponding to the image data can be represented as (6, 256, 2048).

[0057] In the present application, when each computing module processes the received data, if one computing module includes multiple threads, multiple processes, multiple cores, multiple AI chips or multiple servers, for example, one computing module includes multiple AI chips, the multiple AI chips can process the data of one modality in a data parallel, tensor parallel, pipeline parallel or hybrid parallel manner, which is not limited in the present application.

[0058] When the computing module is an AI chip and the multiple AI chips included in the computing system are located in the same computing device, the management module can also perform load balancing on the multiple AI chips included in each computing module. In the case where one computing module includes one or more servers, after each server receives the data processing task sent by the management module, each server can perform load balancing on the multiple AI chips included in the server.

[0059] S205. The target computing module obtains the feature representations obtained by each computing module, concatenates the feature representations corresponding to the data of different modalities, and obtains the fusion features corresponding to the multi-modal data.

[0060] As shown in FIG. 3, after one computing module obtains the feature representation corresponding to the data of one modality, the obtained feature representation is sent to the target computing module. After the target computing module obtains the feature representations corresponding to the data of different modalities, the target computing module concatenates the feature representations corresponding to the data of different modalities to obtain the fusion features corresponding to the multi-modal data, and then inputs the fusion features into the base model for further processing. For example, the multi-modal data described above is multi-modal data input by a user, the base model is used to infer according to the multi-modal data to obtain an inference result, and then the inference result is sent to the management module, and the management module sends the inference result to the user terminal. The target computing module is any one of the multiple computing modules.

[0061] In a possible implementation, the data amount of the fusion features corresponding to the multi-modal data is large, or the parameter amount of the base model is large, and each computing module needs to implement the inference process in a parallel manner. Therefore, after each computing module obtains the feature representation corresponding to the data processed by the computing module, each computing module performs allgather communication with each other, so that each computing module obtains the feature representations corresponding to the data of different modalities. Then, each computing module concatenates the feature representations corresponding to the data of different modalities to obtain the fusion features corresponding to the multi-modal data, and each computing module performs inference according to the fusion features in a data parallel, model parallel or pipeline parallel manner.

[0062] In the present application, since the data of different modalities is processed by different encoders to obtain corresponding feature representations, the dimensions of the feature representations corresponding to the data of different modalities can be different. Referring to FIG. 3, in the present application, before the feature representations corresponding to the data of different modalities are spliced, the feature representations corresponding to the data of different modalities are converted to a unified feature space by an adapter to realize feature alignment, that is, the dimensions of the feature representations corresponding to the data of different modalities are aligned to obtain the aligned feature representations corresponding to the data of each modality. For example, the feature representations corresponding to the data of different modalities will finally be input into a basic model for processing, and the requirement of the input data of the basic model is 2048 dimensions, so the dimensions of the feature representations corresponding to the data of different modalities are converted to 2048 dimensions, and the method of the adapter for realizing feature alignment is not limited in the present application. The above operation of realizing feature alignment can be performed after the feature representations are obtained between the calculation modules, or can be performed after the feature representations corresponding to the data of different modalities are obtained by the above target calculation module, and the present application does not make specific limitation thereon.

[0063] The processing method of multi-modal data provided by the present application is introduced below with a specific application example. In a visual emotion scene, a user uploads a multi-modal data to the above calculation system through a client, the multi-modal data includes a question input in a text manner and a video, wherein the video is a video in which a person A talks to a person B, that is, the video includes audio, and the question is "please analyze the relationship between the two people in the video and the current emotional state of the person wearing white clothes". After receiving the multi-modal data, the management module of the calculation system extracts the audio data from the video by voice extraction technology, so the multi-modal data includes text data, audio data and video data.

[0064] The management module pre-processes the text data, the audio data and the video data through a plurality of CPU cores to obtain pre-processed multi-modal data. Then, according to the sequence length of the data of each modality in the pre-processed multi-modal data, the number of AI chips for processing the data of each modality is determined in the granularity of AI chips. For example, the computing system includes 5 GPUs, and the management module determines that 1 GPU processes the text data, 1 GPU processes the audio data, and the other 3 GPUs process the video data according to the sequence length of the data of each modality, that is, 1 GPU is a computing module for processing text data, 1 GPU is a computing module for processing audio data, and 3 GPUs are computing modules for processing video data. Then, the management module generates three data processing tasks according to the pre-processed data, wherein the first data processing task includes the text data and an operator for processing the text data, the second data processing task includes the audio data and an operator for processing the audio data, and the third data processing task includes the video data and an operator for processing the video data. After the management module generates the three data processing tasks, the three data processing tasks are sent to the corresponding GPUs, respectively. After each GPU receives the data processing task, the data in the data processing task is processed according to the operator in the data processing task to obtain the embedding corresponding to each data, and the obtained embedding is converted to a unified feature space through an adapter to realize feature alignment of data of different modalities. The aligned embedding obtained by each computing module is synchronized again, and the embedding corresponding to the data of different modalities is spliced to obtain the fusion feature corresponding to the multi-modal data. Finally, the fusion feature corresponding to the multi-modal data is input into the pre-trained multi-modal large model, and the pre-trained multi-modal large model outputs an answer to the above problem based on the fusion feature.

[0065] For the above method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the action sequence described, and secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily required by the present application. Those skilled in the art can think of other reasonable combinations of steps based on the above description, which also belongs to the protection scope of the present application. Secondly, those skilled in the art should be familiar with the fact that the embodiments described in the specification all belong to preferred embodiments, and the actions involved are not necessarily required by the present application.

[0066] For the method embodiments described above, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the application is not limited by the order of the described actions, and those skilled in the art should know that the embodiments described in the specification are all preferred embodiments, and the actions involved are not necessarily required by the application. Those skilled in the art can think of other reasonable combinations of steps based on the above description, which are also within the scope of the application. Secondly, those skilled in the art should be familiar with the embodiments described in the specification, which are all preferred embodiments, and the actions involved are not necessarily required by the application.

[0067] The multi-modal data processing method provided by the application is described in detail above in combination with Figures 1 to 3. Next, the multi-modal data processing system and related equipment provided by the application are further introduced in combination with Figures 4-5 respectively.

[0068] Referring to Figure 4, Figure 4 is a multi-modal data processing apparatus provided by the application, which comprises a communication unit 410 and a processing unit 420. The communication unit 410 is configured to obtain multi-modal data, and the multi-modal data comprises at least two of text data, image data, audio data or video data. The processing unit 420 is configured to generate a plurality of data processing tasks according to the multi-modal data, wherein the data of each modality in the multi-modal data corresponds to one data processing task, each data processing task comprises an operator and the data of one modality in the multi-modal data, and each data processing task is configured to instruct a received computing module to encode the data in the task according to the operator in the task. The processing unit 420 distributes each computing task to the corresponding computing module through the communication unit 410, so that each computing module receiving the data processing task executes the data processing task received by itself in parallel, and obtains the feature representation corresponding to the data of different modalities. Each computing module is configured to execute one of the plurality of data processing tasks.

[0069] In a possible implementation, the processing unit is further configured to determine the computing resources allocated to each computing module in the plurality of computing modules according to the data amount of the data of each modality in the multi-modal data, the computing resources being threads, processes, cores of AI chips, AI chips or servers, and each computing module comprising at least one thread, at least one process, at least one core of an AI chip, at least one AI chip or at least one server.

[0070] The processing unit 420 is further configured to preprocess the data of different modalities in the multi-modal data in parallel, and determine the computing resources allocated to the computing module processing each modality data according to the data amount corresponding to the data of each modality.

[0071] Specifically, the operations performed by the multi-modal data processing apparatus can refer to the operations performed by the management module in the embodiment corresponding to FIG. 2, which will not be repeated here.

[0072] Referring to FIG. 5, FIG. 5 is a schematic diagram of a computing device provided by the present application. As shown in FIG. 5, the computing device 500 includes a bus 501, a processor 502, a communication interface 503, and a memory 504. The processor 502, the communication interface 503, and the memory 504 communicate with each other through the bus 501. It should be understood that the present application does not limit the number of processors 502 and memories 504 in the computing device 500.

[0073] The management module described above can be a computing device 500, or a virtual machine (virtual, VM) or a container (Container) deployed in the computing device 500. When the management module is a virtual machine or a container, the operations implemented by the management module are implemented through the software and hardware resources configured to the virtual machine or the container. The computing device 500 can be a server, for example, a server of a cloud data center, an edge server, or a local server in a local data center.

[0074] The bus 501 can be a peripheral component interconnect express (PCIe) bus or an extended industry standard architecture (EISA) bus, a unified bus (Ubus or UB), a compute express link (CXL), a cache coherent interconnect for accelerators (CCIX), etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one line is used in FIG. 5, but it does not mean that there is only one bus or only one type of bus. The bus 501 can include a path for transmitting information between various components of the computing device 500. Among them, the unified bus is, for example, a HyperTransport bus.

[0075] The processor 502 can be a central processing unit (CPU), and can also include a CPU and other hardware chips. The hardware chips can be of various types, for example, can be any one of programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), or digital signal processors (DSPs), etc. The computing device 500 can include one or more hardware chips of any of the above types, or can include multiple types of the above hardware chips, and the embodiments of the present application do not make specific limitations.

[0076] The memory 504 can be a non-volatile memory, for example, a read-only memory (ROM), a programmable ROM (PROM), an erasable PROM (EPROM), an electrically EPROM (EEPROM), or a flash memory. The memory 504 can also be a volatile memory, which can be a random access memory (RAM) used as an external cache. By way of example, but not limitation, many forms of RAM can be used, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). In addition, the memory 504 can also be implemented by a storage class memory (SCM), a phase change memory (PCM), or other types of storage media.

[0077] It should be noted that the same type of storage medium can be configured in the same computing device to implement the memory 504 function, or two or more types of storage media can be configured to implement the memory 504 function, and the present application does not limit this.

[0078] The memory 504 stores executable program code, and the processor 502 executes the executable program code to implement the operations performed by the management module in the embodiment corresponding to FIG. 2.

[0079] The communication interface 503 uses a transceiving module such as but not limited to a transceiver to implement communication between the computing device 500 and other devices or communication networks.

[0080] The computing device 500 described above can further include one or more AI chips 505, that is, the computing device 500 can include the computing system shown in FIG. 1 described above. The processor 502 of the computing device 500 is configured to implement the operations performed by the management module in the embodiment shown in FIG. 2 described above, generate a plurality of data processing tasks, and then distribute the plurality of data processing tasks to each AI chip 505. The one or more AI chips 505 included in the computing device 500 can implement the operations implemented by each computing module in the method embodiment shown in FIG. 2, and details are not repeated here.

[0081] The present application also provides a computer program product containing instructions, which can be software or a program product containing instructions capable of running on a computing device or stored in any available medium. When the computer program product runs on the computing device, it causes the computing device to perform the operations implemented by the management module in the method embodiment corresponding to FIG. 2.

[0082] The present application also provides a computer readable storage medium, which can be any available medium that can be stored by a computing device or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (for example, a floppy disk, a hard disk, a magnetic tape), an optical medium, or a semiconductor medium (for example, a solid state disk), etc. The computer readable storage medium includes instructions that instruct the computing device to perform the operations implemented by the management module in the method embodiment corresponding to FIG. 2.

[0083] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, and not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacements for some technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the protection scope of the technical solutions of the embodiments of the present application.

Claims

1. A multi-modal data processing method, characterized by, A computing system comprising a management module and a plurality of computing modules, comprising: The management module acquires multi-modal data; the multi-modal data comprises at least two of text data, image data, audio data or video data; The management module generates a plurality of data processing tasks according to the multi-modal data, wherein each data processing task comprises data of one modality in the multi-modal data and an operator corresponding to the one modality, the corresponding operator being used to encode the data of the one modality; The management module distributes the plurality of data processing tasks to the plurality of computing modules; wherein each computing module is used to execute one data processing task in the plurality of data processing tasks; The plurality of computing modules execute the respective received data processing tasks in parallel to obtain feature representations corresponding to the data of different modalities; the feature representations corresponding to the data of different modalities are used for training or inference of a multi-modal model.

2. The method of claim 1, wherein, Each computing module in the plurality of computing modules comprises at least one thread, at least one process, at least one core of an AI chip, at least one AI chip or at least one server.

3. The method of claim 2, wherein, Before the management module distributes the plurality of data processing tasks to the plurality of computing modules, further comprising: The management module determines the computing resources allocated to each computing module in the plurality of computing modules according to the data amount of the data of each modality in the multi-modal data, the computing resources being the threads, processes, cores of AI chips, AI chips or servers.

4. The method according to claim 2 or 3, characterized in that, Each computing module executes the received data processing task in a data parallel, model parallel, tensor parallel, pipeline parallel or hybrid parallel manner.

5. The method according to any one of claims 1 to 4, characterized in that, Before the management module distributes the plurality of data processing tasks to the plurality of computing modules, further comprising: The management module pre-processes the data of each modality in the multi-modal data in parallel.

6. The method according to any one of claims 1 to 5, characterized in that, The method further comprises: The plurality of computing modules perform a communication task and synchronize the feature representations corresponding to the data of one modality obtained by executing the respective corresponding data processing tasks; Each computing module in the plurality of computing modules splices the feature representations corresponding to the data of each modality to obtain fused features corresponding to the multi-modal data, the fused features corresponding to the multi-modal data being used for training or inference of a multi-modal model.

7. A computing system, comprising: The computing system comprises a management module and a plurality of computing modules, the management module being used to perform the operations performed by the management module in any one of claims 1-6, and the plurality of computing modules being used to perform the operations performed by the computing module in any one of claims 1-6.

8. A management module, characterized by A computing system comprising a management module and a plurality of computing modules, the computing system being used to process multi-modal data; the management module comprising a communication unit and a processing unit, wherein, The communication unit is used to acquire multi-modal data; the multi-modal data comprises at least two of text data, image data, audio data or video data; The processing unit is used to generate a plurality of data processing tasks according to the multi-modal data, wherein each data processing task comprises data of one modality in the multi-modal data and an operator corresponding to the one modality, the corresponding operator being used to encode the data of the one modality; The processing unit is configured to generate a plurality of data processing tasks according to the multi-modal data, wherein each data processing task comprises data of one modality in the multi-modal data and an operator corresponding to the one modality, and the corresponding operator is configured to encode the data of the one modality. The communication unit is further configured to distribute the plurality of computing tasks to the plurality of computing modules, so that the plurality of computing modules perform the data processing tasks received respectively in parallel to obtain feature representations corresponding to data of different modalities; each computing module is configured to perform one data processing task in the plurality of data processing tasks, and the feature representations corresponding to the data of different modalities are used for training or inference of a multi-modal model.

9. The management module of claim 8, wherein, The processing unit is further configured to determine computing resources allocated to each computing module in the plurality of computing modules according to a data amount of data of each modality in the multi-modal data, the computing resources being threads, processes, cores of AI chips, AI chips, or servers, and each computing module comprising at least one thread, at least one process, at least one core of an AI chip, at least one AI chip, or at least one server.

10. The management module according to claim 8 or 9, characterized in that, The processing unit is further configured to preprocess the data of each modality in the multi-modal data in parallel.

11. A computer readable storage medium characterized by, The computer program instructions, when executed by a computing device, cause the computing device to perform operations performed by the management module in any one of claims 8 to 10.

12. A computer program product comprising instructions, characterized in that, The computing device, when the instructions are executed by the computing device, implements operations performed by the management module in any one of claims 8 to 10.

Images