Multi-modal task processing method and system, multi-modal dialogue task processing method and system, and device

Abstract

Provided in the present application are a multi-modal task processing method and system, a multi-modal dialogue task processing method and system, and a device. The methods in the present application can be applied to a task processing scenario based on a large model, and by means of obtaining a target task model after the computational efficiency of an operator of a model unit in an initial task model is optimized by using an optimization strategy, the computational efficiency of the large model is improved. When a multi-modal task is processed, multi-modal input data is input into a target task model, and the multi-modal input data is coded and fused by means of a target task model, thereby obtaining a multi-modal fused feature representation; and task processing is executed on the basis of the multi-modal fused feature representation, and a task processing result is generated. An inference mode capable of accepting a multi-modal input is implemented, and on the basis of a target task model that is obtained after optimization of the computational efficiency, the computational efficiency of the model is improved while maintaining the accuracy of the model, such that the time consumed for multi-modal task processing is greatly shortened, thereby improving the efficiency and performance of multi-modal task processing.

Description

Method, system and device for multimodal task processing and dialogue task processing

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on August 2, 2023, with application number 202310969749.6 and application name “Methods, systems and devices for multimodal task processing and dialogue task processing”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to computer technology, and in particular to a method, system, and device for multimodal task processing and dialogue task processing. Background Art

[0003] With the rise and widespread application of natural language processing (NLP), a large number of natural language generation (NLG) models have emerged in the industry. These models are enabling the automation of an increasing number of work and learning tasks.

[0004] Traditional natural language generation (NLG) models typically use text as input and output, and their encoding modules only support text encoding. With the advancement of technology, text alone is no longer sufficient to demonstrate a model's capabilities. An increasing number of models are using multimodal data, such as images, audio, and video, for training and reasoning. This ultimately enables models to understand multimodal data, enabling them to complete multimodal tasks such as "speaking from images" and "summarizing video content."

[0005] However, this type of model still has performance issues when performing multimodal reasoning. Multimodal data takes up a large amount of input length. For example, after encoding, an image often takes up an input length equivalent to hundreds of words, which greatly affects the overall reasoning performance of the model, resulting in long reasoning time and low performance of multimodal tasks.

[0006] Summary of the Invention

[0007] The present application provides a method, system, and device for multimodal task processing and dialogue task processing to solve the problems of long reasoning time and low performance of multimodal tasks.

[0008] In a first aspect, the present application provides a multimodal task processing method, comprising:

[0009] Acquire multimodal input data to be processed; input the multimodal input data into a target task model, encode and fuse the multimodal input data through the target task model to obtain a feature representation of multimodal fusion, and perform task processing according to the feature representation of multimodal fusion to generate a task processing result; wherein the target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy.

[0010] In a second aspect, the present application provides a multimodal dialogue task processing method, applied to a server, comprising:

[0011] A dialogue task request is received from an end-side device, wherein the dialogue request includes multimodal input data; the multimodal input data is input into a target dialogue macromodel, the multimodal input data is encoded and fused through the target dialogue macromodel to obtain a feature representation of the multimodal fusion, and dialogue task processing is performed according to the feature representation of the multimodal fusion to generate a reply text; wherein the target dialogue macromodel is obtained by optimizing the computational efficiency of the operators of the model units in the initial dialogue macromodel using an optimization strategy; and the reply text is sent to the end-side device.

[0012] In a third aspect, the present application provides a multimodal dialogue task processing system, including a terminal device and a server.

[0013] The end-side device is used to receive a dialogue task request sent by the end-side device and send multimodal input data to the server, wherein the dialogue task request includes the multimodal input data;

[0014] The server is configured to receive multimodal input data, input the multimodal input data into a target dialogue macromodel, encode and fuse the multimodal input data using the target dialogue macromodel to obtain a multimodal fused feature representation, and perform dialogue task processing based on the multimodal fused feature representation to generate a reply text; wherein the target dialogue macromodel is obtained by optimizing the computational efficiency of operators of model units in the initial dialogue macromodel using an optimization strategy;

[0015] The server is further configured to send the reply text to the terminal device;

[0016] The terminal-side device is also used to receive and output the reply text.

[0017] In a fourth aspect, the present application provides a multimodal task processing system, including a terminal device and a server,

[0018] The terminal device is used to receive multimodal input data to be processed input by the user and send the multimodal input data to the server;

[0019] The server is configured to receive multimodal input data sent by an end-side device, input the multimodal input data into a target task model, encode and fuse the multimodal input data through the target task model to obtain a multimodal fused feature representation, and perform task processing based on the multimodal fused feature representation to generate a task processing result; wherein the target task model is obtained by optimizing the computational efficiency of operators of model units in the initial task model using an optimization strategy;

[0020] The server is further configured to send the task processing result to the terminal device;

[0021] The terminal side device is further configured to receive a task processing result and output the task processing result.

[0022] In a fifth aspect, the present application provides a server comprising: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method described in the first aspect or the second aspect.

[0023] In a sixth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by a processor, they are used to implement the method described in the first aspect or the second aspect.

[0024] The method, system and equipment for multimodal task processing and dialogue task processing provided in the present application obtain a target task model by optimizing the computational efficiency of the operators of the model units in the initial task model through an optimization strategy. When performing multimodal task processing, the multimodal input data to be processed is obtained, the multimodal input data is input into the target task model, the multimodal input data is encoded and fused through the target task model to obtain a feature representation of the multimodal fusion, and task processing is performed according to the feature representation of the multimodal fusion to generate a task processing result, thereby realizing an inference mode that can accept multimodal input. Moreover, since the target task model used to perform multimodal task processing is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model through an optimization strategy, the computational efficiency of the model can be improved while maintaining the accuracy of the model from being degraded, thereby greatly shortening the time consumption of multimodal task processing and improving the efficiency and performance of multimodal task processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0026] FIG1 is a schematic diagram of an exemplary task processing system architecture applicable to the present application;

[0027] FIG2 is a flow chart of a multimodal task processing method provided by an exemplary embodiment of the present application;

[0028] FIG3 is a flow chart of a method for multimodal coding fusion provided by an exemplary embodiment of the present application;

[0029] FIG4 is a flow chart of a method for optimizing the computational efficiency of a model provided by an exemplary embodiment of the present application;

[0030] FIG5 is an architecture diagram of a multimodal task processing method provided by an exemplary embodiment of the present application;

[0031] FIG6 is an interaction flow chart of a multimodal task processing system provided by an exemplary embodiment of the present application;

[0032] FIG7 is a flow chart of a multimodal dialogue task processing method provided by an exemplary embodiment of the present application;

[0033] FIG8 is an interaction flow chart of a multimodal dialogue task processing system provided by an exemplary embodiment of the present application;

[0034] FIG9 is a schematic structural diagram of a multimodal task processing device provided by an exemplary embodiment of the present application;

[0035] FIG10 is a schematic diagram of the structure of a server provided in an embodiment of the present application.

[0036] The above drawings illustrate specific embodiments of the present application, which will be described in more detail below. These drawings and the textual description are not intended to limit the scope of the present application in any way, but rather to illustrate the concepts of the present application to those skilled in the art by reference to specific embodiments. DETAILED DESCRIPTION

[0037] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0038] First, let’s explain the terms involved in this application:

[0039] Multimodality: Multiple heterogeneous modal data, including but not limited to: text, images, video, audio, program code, etc.

[0040] MatMul or GEMM (General Matrix Multiplication): matrix multiplication.

[0041] LayerNorm: Normalize each single batch.

[0042] GELU: Gaussian Error Linear Unit activation function.

[0043] Embedding: Use a vector to represent a word.

[0044] Int8 quantization technology: supports static quantization, training-aware quantization, and dynamic quantization. Dynamic quantization is recommended because it achieves lossless accuracy and accelerates performance without fine-tuning by algorithm developers.

[0045] Subgraph Fuse: Subgraphs automatically match the user's computational graph to replace fusion operators, thereby merging the memory access of many element-wise operators (such as +-* / ) in the network. Element-wise operators refer to operations that need to be run bit by bit, and have a wide range of element-to-element calculations, such as element addition, element dot multiplication, element subtraction, and extreme value extraction.

[0046] Operator: An operator is a computational operation performed on data. Operators are implemented as functions based on specified parameters. Different operators are connected to each other, representing the data transmission relationship between them. Each operator can perform a corresponding computation on the input data and output the corresponding calculation result. For example, there are operators representing basic arithmetic operations such as addition, subtraction, multiplication, and division; operators representing matrix transposition operations; operators representing decoding operations; and operators representing operations such as image filtering and edge detection in image processing.

[0047] Operator Optimization (OPTune): Key operators usually have multiple implementations (for example, Conv has more than 10 implementations). An automatic performance evaluation is performed at runtime, and the fastest implementation is selected for inference.

[0048] Global Optimization (Global Opt.): Use dynamic programming to calculate the optimal data format and operator combination in the network computing process, eliminating certain repeated calculations and repeated memory accesses.

[0049] Multimodal tasks: refers to downstream tasks whose input and output data involve multiple modal data such as images and text, such as visual question answering tasks, image description tasks, visual implication tasks, referential expression and understanding tasks, image generation tasks, etc.

[0050] Visual question answering task: Given an input image and a question, determine the answer to the question from the visual information of the input image.

[0051] Image description task: Generate description text for the input image.

[0052] Visual entailment task: predict the semantic relevance of an input image and text, i.e., entailment, neutrality, or contradiction.

[0053] Referential expression and comprehension task: locate the image area corresponding to the input text in the input image based on the input text.

[0054] Image generation task: Generate an image based on the input description text.

[0055] Text-based sentiment classification task: predict the sentiment classification information of the input text.

[0056] Text summarization task: Generate summary information of the input text.

[0057] Multimodal pre-trained model: refers to a pre-trained model whose input and output data involve multiple modal data such as images and text. After fine-tuning and training, it can be applied to multimodal task processing.

[0058] Pre-trained language model: A pre-trained model obtained by pre-training a large-scale language model (LLM).

[0059] Natural Language Generation (NLG): It is an automated process of generating language text by computers under specific interactive goals. Its main purpose is to automatically construct high-quality language text that can be understood by humans.

[0060] Large models refer to deep learning models with large-scale model parameters, typically containing hundreds of millions, tens of billions, or even hundreds of billions of model parameters. Large models, also known as foundation models, are pre-trained on large-scale unlabeled corpora, producing pre-trained models with parameters exceeding 100 million. Such models are adaptable to a wide range of downstream tasks and exhibit good generalization capabilities, such as large language models (LLMs) and multi-modal pre-training models.

[0061] When large models are used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. Large models can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, they can be applied to computer vision tasks such as visual question answering (VQA), image caption (IC), and image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0062] In recent years, natural language generation models have become a key area of ​​artificial intelligence and natural language processing. Compared to traditional statistical language models, modern language generation models based on deep learning models mostly use end-to-end learning to build relationships between vocabulary and context, significantly improving the performance of generated text. Transformer-based pre-trained models, in particular, are able to effectively capture linguistic knowledge and have significantly advanced the development of natural language generation models. Natural language generation models can be implemented using large models, including multimodal pre-trained models and pre-trained language models.

[0063] Traditional natural language generation (NLG) models typically use text as input and output, and their encoding modules only support text encoding. With the advancement of technology, text alone is no longer sufficient to demonstrate a model's capabilities. More and more models are using multimodal data, such as images, audio, video, and program code, for training and reasoning. This ultimately enables models to understand multimodal data, enabling them to complete multimodal tasks such as "speaking from images" and "summarizing video content."

[0064] However, this type of model still has performance issues when performing multimodal reasoning. Multimodal data takes up a large amount of input length. For example, after encoding, an image often takes up an input length equivalent to hundreds of words, which greatly affects the overall reasoning performance of the model, resulting in long reasoning time and low performance of multimodal tasks.

[0065] The present application provides a multimodal task processing method, which uses an optimization strategy to optimize the computational efficiency of the operators of the model units in the initial task model to obtain a target task model. When performing multimodal task processing, the multimodal input data to be processed is obtained, the multimodal input data is input into the target task model, the multimodal input data is encoded and fused through the target task model to obtain a feature representation of the multimodal fusion, and task processing is performed based on the feature representation of the multimodal fusion to generate a task processing result, thereby realizing an inference mode that can accept multimodal input. Moreover, since the target task model used to perform multimodal task processing is obtained by using an optimization strategy to optimize the computational efficiency of the operators of the model units in the initial task model, the computational efficiency of the model can be improved while maintaining the accuracy of the model from being degraded, thereby greatly shortening the time consumption of multimodal task processing and improving the efficiency and performance of multimodal task processing.

[0066] Figure 1 is a schematic diagram of an example task processing system architecture applicable to this application. As shown in Figure 1, the system architecture includes a server and a terminal device. A communication link is provided between the server and the terminal device, enabling communication between the server and the terminal device.

[0067] Among them, the server can be a server cluster deployed in the cloud, or a local device with computing power. The server runs a target task model for performing multimodal tasks. The target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy. The target task model not only maintains the reasoning accuracy of the initial task model, but also greatly improves the computational efficiency of the model and shortens the time consumed in multimodal task reasoning. Among them, the target person model can use a natural language generation (NLG) model, or it can be various pre-trained large models, such as pre-trained language large models, multimodal pre-trained models, etc., which are not specifically limited here.

[0068] When executing a multimodal task, the server is used to obtain the multimodal input data to be processed, input the multimodal input data into the target task model, encode and fuse the multimodal input data through the target task model, obtain a multimodal fused feature representation, and perform task processing based on the multimodal fused feature representation to generate a task processing result. The feature representation is obtained by encoding the input data, or by fusing it after encoding, and is typically a tensor representation, but can also be a vector representation or other representation of the encoding result, and is not specifically limited here.

[0069] The end-side device is an electronic device used by the user. Specifically, it can be a hardware device with network communication functions, computing functions, and information display functions, including but not limited to smartphones, tablets, desktop computers, servers, etc. The user refers to the individual or organization requesting the execution of a multimodal task. The user provides the server with multimodal input data to be processed through the end-side device and obtains the task processing results from the server. Furthermore, the end-side device outputs the task processing results, or performs subsequent processing logic based on the task processing results to realize various human-computer interaction functions.

[0070] In one example scenario, this solution is applied to a multimodal conversation / question-and-answer scenario, enabling human-computer conversation / question-and-answer based on a target conversation model. The target conversation model can use a natural language generation (NLG) model, a pre-trained language model, a multimodal pre-trained model, or the like, without specific limitations. Users input multimodal input data, including but not limited to text, images, audio, video, and program code, through a client-side device. The client-side device sends the multimodal input data to a server. The server inputs the multimodal input data into a target task model, which encodes and fuses the multimodal input data to obtain a multimodal fused feature representation. Task processing is then performed based on the multimodal fused feature representation, generating a task processing result. Furthermore, the server sends the task processing result to the client-side device. The client-side device outputs the task processing result to enable human-computer interaction with the user. Furthermore, the client-side device can perform subsequent processing on the task processing result based on configured demand logic and output the subsequent processing result to meet user needs and enable various human-computer interaction functions with the user.

[0071] In an optional embodiment, after generating the task processing results, the server can also execute the post-processing logic based on the configured post-processing logic to obtain the final output result, and send the final output result to the end-side device. The end-side device outputs the final output result to meet user needs and realize various human-computer interaction functions with the user.

[0072] It should be noted that the method of the present application can be applied to various tasks in the fields of computer vision, natural language processing, and other fields such as machine translation, visual question answering, image description, and text summary generation. It can be applied to application scenarios such as digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, and open source cloud platform management services (MaaS, Metal as a Service). When applied to different tasks, the input modalities contained in the multimodal input data can be different and can be specifically configured according to the needs of the actual task scenario. This embodiment does not specifically limit the tasks and application scenarios used.

[0073] The following specific embodiments describe in detail the technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0074] FIG2 is a flowchart of a multimodal task processing method provided by an exemplary embodiment of the present application. The execution subject of this embodiment is the server in the aforementioned system architecture. As shown in FIG2, the specific steps of the method are as follows:

[0075] Step S21: Obtain multimodal input data to be processed.

[0076] Multimodal input data includes, but is not limited to, data in the following modalities: text, images, audio, video, and program code. When applied to different task scenarios, the input data to be processed may include data in different modalities.

[0077] For example, in a human-computer dialogue task scenario, the multimodal input data provided by the user includes two inputs: a given image and a question text. The question text may be something like "Who is the person in the picture?" The target task model is used to perform task processing based on the multimodal input data. The task processing result may be a response text, which serves as a response to the question text.

[0078] For example, in a text summarization task scenario, the multimodal input data input by the user includes a text paragraph. The target task model is used to perform task processing based on the multimodal input data, and the task processing result obtained is the summary information of the generated text paragraph.

[0079] In practical applications, there are various ways for the server to obtain the multimodal input data to be processed in this step, and the specific method is selected based on actual circumstances. This embodiment does not impose any limitations on this. For example, the server can receive the multimodal input data input by the user, or the user can store the multimodal input data in advance in a data storage unit of the data input server, and the server can extract the multimodal input data to be processed from the data storage unit.

[0080] In addition, the server can process multimodal input data for a single task request, or it can process multimodal input data for multiple task requests simultaneously, without specific limitations here. The multimodal input data for a task request can include one or more inputs, and different inputs can be data of different modalities. In some scenarios, multimodal input data can also include multiple inputs of the same modality, such as multiple images, multiple paragraphs of text with different physical meanings, etc.

[0081] Step S22: input the multimodal input data into the target task model, encode and fuse the multimodal input data through the target task model to obtain a feature representation of the multimodal fusion, and perform task processing according to the feature representation of the multimodal fusion to generate a task processing result; wherein, the target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy.

[0082] In this embodiment, the target task model can receive multimodal input data, which includes multiple inputs, and any two inputs can be data of different or same modalities.

[0083] After obtaining the multimodal input data input by the user, the multimodal input data is input into the target task model, and the multimodal input data is encoded and fused through the target task model to obtain the multimodal fusion feature representation, and task processing is performed according to the multimodal fusion feature representation to generate the task processing result, thereby realizing multimodal task processing.

[0084] For example, in the aforementioned human-computer dialogue task scenario, the multimodal input data provided by the user includes a given image and a question text. The given image and question text are input into the target task model. The target task model then encodes and fuses the given image and question text, respectively, to obtain a multimodal fused feature representation of the image and text. Furthermore, reasoning is performed based on the multimodal fused feature representation to implement multimodal task processing, generate a response text to the question text, and obtain the task processing result.

[0085] It should be noted that, when the previous multimodal input data only contains text input, that is, there is no non-text modal input in the multimodal input data, the solution of this embodiment can also be used to directly input the text input into the target task model, encode the text input through the target task model, obtain the feature representation of the text input, and perform task processing based on the feature representation of the text input to generate the task processing result.

[0086] The solution of this embodiment is not limited to the type and representation of multimodal input data. Whether it is existing pictures, videos, voice, or more data types / modalities that may appear in the future, this reasoning process can be used to accelerate model reasoning, and it has strong versatility.

[0087] In this embodiment, the target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy. Specifically, the target task model can be obtained by updating and setting the operators of each model unit based on the operator information of each model unit in the initial task model using a corresponding computational efficiency improvement strategy. Since the target task model is obtained by updating and setting the operators of each model unit based on the operator information of each model unit in the initial task model using a corresponding computational efficiency improvement strategy, the computational efficiency of the model can be improved while maintaining the original accuracy of the initial task model from being degraded, thereby greatly shortening the task processing time and improving the efficiency and performance of task processing. Among them, the computational efficiency improvement strategies corresponding to different types of operators may be different.

[0088] Exemplarily, for operators of arithmetic operation type in each model unit, multiple operators of arithmetic operation type in the same model unit may be fused to construct a target operator.

[0089] Exemplarily, for the transposition operation type operator in each model unit, a target operator can be constructed according to a pre-set stride parameter. The target operator is used to store target feature data in memory according to the stride parameter, and read the target feature data from the memory when calculating the target feature data. The target feature data is the feature data for the transposition operation.

[0090] For example, for activation type operators in each model unit, the activation type operators can be placed after the execution process of the target operator, and the execution logic of the target operator can be updated, which can reduce the overhead of launching the kernel and achieve a balance between accuracy and speed.

[0091] Exemplarily, for the decoding type operator in each model unit, the decoding result corresponding to the decoding type operator can be obtained; the decoding result is stored in a pre-set decoding cache unit; according to the decoding result in the decoding cache unit, the decoding logic of the operator is updated and set to obtain the target operator, which can avoid unnecessary repeated calculations in the decoding process and improve the model operation speed.

[0092] In an optional embodiment, in step S22, when encoding and fusing the multimodal input data using the target task model to obtain a multimodal fused feature representation, each input in the multimodal input data may be encoded separately to obtain a feature representation corresponding to each input. Furthermore, the feature representations corresponding to each input are concatenated to achieve fusion of the feature representations of inputs of different modalities, thereby obtaining a multimodal fused feature representation.

[0093] In another optional embodiment, as shown in FIG3 , in the aforementioned step S22 , the multimodal input data is encoded and fused through the target task model to obtain a multimodal fused feature representation, which can be specifically implemented by the following steps S31 to S34:

[0094] Step S31: Encode the non-text modal inputs contained in the multimodal input data respectively to obtain feature representations of the non-text modal inputs.

[0095] In this step, the non-text modal inputs in the multimodal input data are encoded separately to obtain feature representations of each non-text input.

[0096] Different encoding algorithms / models can be used to encode inputs of different modalities. For example, for pictures in multimodal input data, the image encoding model is used to encode the input pictures separately to obtain a feature representation of the input pictures. For audio in multimodal input data, the audio encoding model / algorithm is used to encode the input audio separately to obtain a feature representation of the input audio. For video in multimodal input data, the video encoding model / algorithm is used to encode the input video separately to obtain a feature representation of the input video. Specifically, the encoding model / algorithm of the corresponding modality can be selected according to the modality of each input. For the encoding model / algorithm used for any modality, the encoding model / algorithm commonly used for encoding the modal data can be selected based on experience, and no specific limitation is made here.

[0097] In actual applications, the server can process the multimodal input data of a task request, or it can process the multimodal input data of multiple task requests at the same time, which is not specifically limited here. The multimodal input data of a task request can include one or more inputs, and different inputs can be data of different modalities. In some scenarios, multimodal input data can also include multiple inputs of the same modality, for example, a task request can input multiple pictures, or multiple paragraphs of text with different physical meanings, etc.

[0098] In an optional embodiment, a modal identification information can be set for each modality to distinguish and refer to different modalities. For example, different modalities can be numbered and the corresponding numbers of each modality can be used as modal identification information. For example, the number of the picture can be set to 10001, the number of the video can be set to 10002, the number of the audio can be set to 10003, and so on. In addition, the modal identification information of each modality can also be represented by different characters / strings, and the modal identification information of each modality can be set according to the specific application scenario and experience, and is not specifically limited here.

[0099] After obtaining the feature representation of the non-text modality input, a sequence consisting of the feature representation of at least one input of the same non-text modality in the multimodal input data is stored in correspondence with the modality identification information corresponding to the non-text modality.

[0100] For example, assuming that the multimodal input data includes text A, picture B, picture C, audio D, and audio E, the picture number can be set to 10001, and the audio number can be set to 10003. Then, after encoding picture B and picture C respectively to obtain the feature representation of picture B and the feature representation of picture C, and encoding audio D and audio E respectively to obtain the feature representations corresponding to audio D and audio E, for the picture modality, the feature representations of the two pictures B and C are arranged into a feature representation sequence of the picture in the order in which they appear in the multimodal input data, and the feature representation sequence of the picture is stored in correspondence with the modal identification information 10001 corresponding to the picture; for the audio modality, the feature representations of the two audios D and E are arranged into a feature representation sequence of the audio in the order in which they appear in the multimodal input data, and the feature representation sequence of the audio is stored in correspondence with the modal identification information 10003 corresponding to the audio, resulting in the storage content shown in Table 1 below.

[0101] Table 1

[0102] Optionally, the sequence of feature representations of any modality may be stored in a list format.

[0103] By storing the sequence of feature representations of the input of each modality in correspondence with the modality identification information, and by traversing the feature representations in the feature representation sequence corresponding to each modality, the replacement of the feature representation in the subsequent step S34 is achieved, thereby improving the processing efficiency of obtaining the feature representation of multimodal fusion.

[0104] Step S32: Replace the non-text modal input in the multimodal input data with the corresponding modality identification information to obtain a plain text input.

[0105] In this step, the non-text modal input in the original multimodal input data is replaced with the modal identification information of the corresponding modality, and the replaced input data only contains data of a single text modality, obtaining a pure text input.

[0106] Additionally, if the multimodal input data for the same task request includes multiple inputs of a certain modality, each input of that modality will be replaced with the modality identification information of that modality. For example, if a multimodal input data contains two or more input images, the modality identification information of these images will be replaced in sequence.

[0107] For example, the non-text modal input in the multimodal input data is replaced with a text segment of the same length, spliced ​​together using the corresponding modal identification information, based on the length of the non-text modal input, to obtain the plain text input. This maintains the length of the original input, facilitating the replacement of the feature representation in the subsequent step S34, and improving the efficiency of obtaining the multimodal fusion feature representation.

[0108] For example, a modality number is used as the modality identification information, and each modality number occupies one character. Assume that the image number can be 10001. For an image in the multimodal input data, if an image occupies 384 characters, the image in the multimodal input data is replaced with 384 10001s.

[0109] In addition, for images in multimodal input data, when encoding the image data, a preset encoding interface can be used to convert the Vision Transformer (VIT) model used for image encoding into an Open Neural Network Exchange (ONNX) model through the model conversion service, which can improve the efficiency and performance of image encoding.

[0110] The input encoding process is divided into character encoding and position encoding. Character encoding converts input data from a single symbol (such as a character in text or a pixel block in an image) into a corresponding feature representation (tensor representation). This process itself has relatively stable performance. Position encoding calculates a tensor offset based on the position of the character in the input and adds it to the feature representation (tensor representation) after character encoding.

[0111] Step S33: Encode the plain text input to obtain a feature representation of the plain text input.

[0112] In this step, the plain text input is encoded using a text encoding model / algorithm to obtain a feature representation of the plain text input. The text encoding model / algorithm used can be implemented using an efficient algorithm based on the needs of the actual application scenario and is not specifically limited here.

[0113] In this step, you can also encode the plain text input by calling the preset efficient encoding interface, thereby maintaining the performance and efficiency of model reasoning.

[0114] Step S34: Use the feature representation of the non-text modal input to replace the feature representation of the corresponding modality identification information in the feature representation of the plain text input to obtain a multimodal fusion feature representation.

[0115] In order to achieve the fusion of feature representations of multimodal input, in this step, based on the feature representation of plain text input, the feature representation of non-text modal input is used to replace the feature representation of the corresponding modality identification information in the feature representation of plain text input to obtain a multimodal fused feature representation.

[0116] Specifically, for any feature representation of non-text modal input, based on the length of the feature representation of the non-text modal input, a text fragment of the same length spliced ​​together with the corresponding modal identification information is searched in the plain text input to determine the position of the text fragment in the plain text input; based on the position of the text fragment in the plain text input, the feature representation of the non-text modal input is used to replace the content at the corresponding position in the feature representation of the plain text input to obtain a multimodal fused feature representation.

[0117] In addition, in this step, before replacing the feature representation of non-text modal input, it is possible to check whether there is a feature representation of non-text modal input. If there is no feature representation of non-text modal input, the feature representation of the plain text input is directly used as the final feature representation to perform subsequent task processing and obtain the task processing result.

[0118] If there is a feature representation of non-text modal input, then all feature representations of non-text modal input are traversed, and the modal identification information of each modality corresponds to the stored feature representation of the modal input. If the multimodal input data includes multiple inputs of a certain modality, that is, the modal identification information of the modality corresponds to multiple feature representations, then the feature representation of the modal input is parsed in turn, and the first text segment spliced ​​with the corresponding modal identification information is searched in the plain text input to determine whether the length of the text segment spliced ​​with the modal identification information is the same as the length of the feature representation of the modal input. If the same, the feature representation of the modal input is used to replace the text segment, and then the next feature representation is replaced until the feature representations of all non-text modal inputs are replaced, and a multimodal fusion feature representation is obtained.

[0119] If the text segment formed by the corresponding modal identification information is not found in the plain text input, or the length of the text segment formed by the found modal identification information is different from the length of the feature representation of the modal input, an error message can be output to indicate that the task execution has failed.

[0120] In an optional embodiment, a multimodal fusion operator can be used to implement the use of feature representations of non-text modal inputs to replace the feature representations of the corresponding modal identification information in the feature representations of plain text inputs, thereby obtaining a multimodal fusion feature representation. Specifically, the feature representations of all non-text modal inputs, the replaced plain text inputs, and the feature representations of the plain text inputs are passed into the multimodal fusion operator, and the multimodal fusion operator uses the feature representations of non-text modal inputs to replace the feature representations of the corresponding modal identification information in the feature representations of the plain text inputs, thereby obtaining a multimodal fusion feature representation. The multimodal fusion operator outputs the multimodal fusion feature representation. In addition, in the case where the feature representation of the non-text modal input is not passed in, the multimodal fusion operator directly outputs the feature representation of the plain text input. In the case where there is no feature representation of the non-text modal input, that is, in the case where there is no non-text modal input, the feature representation of the plain text input is also the feature representation of the original input text.

[0121] In real-world scenarios like online conversations, the emergence of multimodality is often unpredictable. For example, a user might suddenly insert an image during a chat. Therefore, multimodal understanding must be routinely supported by the inference engine. The advantage of this solution is that it can routinely support multimodal input while maintaining the same inference performance for pure text input scenarios. In other words, introducing multimodality does not affect the original pure text inference performance.

[0122] The method of this embodiment encodes the non-text modal input contained in the multimodal input data separately to obtain the feature representation of the non-text modal input; replaces the non-text modal input in the multimodal input data with the corresponding modal identification information to obtain the plain text input; encodes the plain text input to obtain the feature representation of the plain text input; uses the feature representation of the non-text modal input to replace the feature representation of the corresponding modal identification information in the feature representation of the plain text input to obtain the feature representation of multimodal fusion, thereby realizing the fusion of multimodal input. It can support scenarios where the input data only contains text content but does not contain non-text modal content, realize efficient processing of multimodal tasks, and retain the pure text reasoning performance of the model.

[0123] Considering that in practical applications, the input encoding process is divided into character encoding and position encoding. The character encoding process calculates the corresponding feature representation (tensor representation) of the input data from a single symbol (such as a character in text or a pixel block in an image). The performance of this process itself is relatively stable. Position encoding calculates a tensor offset based on the position of the character in the input and adds it to the feature representation (tensor representation) after character encoding.

[0124] As the number of user task requests increases, the inference engine cannot fully utilize the hardware performance by processing only one request at a time. Therefore, multiple requests will be grouped together for simultaneous inference. In the process of grouping requests, it is inevitable that the input lengths of different requests will be different.

[0125] Current model inference solutions (such as the PyTorch model) require padding placeholders to the end of shorter inputs, ensuring that all requests have the same input length as the longest one. This aligns the input lengths of all requests before encoding and inference can proceed. This approach can lead to a large amount of invalid placeholder padding and computation when input lengths within a group of requests vary significantly. For example, if a request with an input length of 20 is grouped with a request with an input length of 1000 and padded with placeholders, the encoding process is actually performing the same encoding process for two requests with an input length of 1000, wasting significant computing resources. There are also issues with positional encoding calculations. For example, in a single inference run, the position of the next generated word for a request with an input length of 20 should be 21. However, in existing model inference solutions, due to placeholder padding and uniform length, the position encoding of subsequent generated characters starts at 1001. This leads to inconsistent encoding results between single and combined runs, affecting the stability of text generation.

[0126] In an optional embodiment, the multimodal input data to be processed includes multimodal input data of multiple task requests, and the multiple task requests constitute a group of requests. When encoding the plain text input in the aforementioned step S33, the plain text input of each task request is passed to a preset encoding interface, and the maximum length of the plain text input of each task request is adjusted through the preset encoding interface. By filling a placeholder at the end of the plain text input whose length is less than the maximum length, the lengths of the plain text inputs of the multiple task requests are aligned, and the aligned plain text inputs of the multiple task requests are encoded. Among them, the placeholder can be selected according to the actual application scenario and experience, and is not specifically limited here.

[0127] Furthermore, when encoding the aligned plain text input of multiple task requests, the placeholders contained in the aligned plain text input are parsed to determine the actual effective length of the aligned plain text input and the placeholder offset of the placeholders contained therein; based on the actual effective length of the aligned plain text input and the placeholder offset of the placeholders contained therein, the actual effective input in the aligned plain text input of multiple task requests is encoded.

[0128] Specifically, the plain text inputs of a group of requests are parsed separately, and the placeholder padding of each request's plain text input is scanned to calculate the actual effective length of each request's plain text input (the number of non-placeholder symbols), and the placeholder offset of each request's plain text input is obtained. The placeholder offset of the aligned plain text input is the actual length of the plain text input minus the unified length of the plain text input of the group of requests after padding. The placeholder offset of the plain text input is a negative number or zero.

[0129] Furthermore, during the encoding process, the feature representation corresponding to each character in the aligned plain text input is calculated in sequence. During the execution of the calculation kernel, it is possible to determine which input the symbol currently being calculated belongs to. If the symbol position is greater than the actual effective length of the input, no calculation is performed and the feature representation of the character is padded with zeros.

[0130] Furthermore, during the subsequent inference process of the model, new characters are continuously generated, and these new characters also need to be encoded. The encoding process involves the calculation of position encoding. For each newly generated character, its actual position is the current position plus the placeholder offset.

[0131] For example, consider a request with a plain text input length of 20 and a plain text input length of 1000. After alignment, the plain text inputs have the same length of 1000. For the plain text input with a length of 20, the placeholder offset is -980. When calculating the position encoding of the first newly generated character, the current position is 1001, and the placeholder offset -980 is added to obtain the final actual position of 21. This result is the same as the result of encoding the request alone, ensuring the stability of the text generation effect.

[0132] The method of this embodiment, when encoding multimodal input data, uses placeholders to fill request inputs of different lengths to a uniform length through a preset encoding interface. However, in the actual encoding process, encoding is performed according to the actual effective length of each request input. This method can avoid invalid calculations of filling placeholders during the encoding process while processing multiple task request inputs in parallel, thereby improving the encoding efficiency and performance in multi-task request scenarios, thereby improving the overall efficiency and performance of task reasoning.

[0133] In an optional embodiment, when constructing the target task model, it can be divided into the following three steps:

[0134] Model format conversion and loading: Obtain the initial task model, parse the initial task model through the model converter (Model Convertor), and convert the initial task model into an internal representation format that can be processed by the server, such as .as files and .asparam files. Among them, the .as file is used to store the model representation graph structure, and the .asparam file is used to store specific parameters, that is, operator weights. Furthermore, the model parser (Model Parser) can be used to read the initial task model file in the internal representation format, parse the initial task model file, obtain the model protocol (Model Protocol), and generate a model instance (Model Instance) according to the configuration protocol (Config Protocol) in the model protocol. Using each operator in the model instance as a node, construct the model representation graph corresponding to the initial task model, and identify the operator information of each operator in the model representation graph.

[0135] Model optimization: The Graph Optimizer adjusts the model instance and generates an optimized internal representation of the model (AsModel). This means that the graph optimizer uses the computational efficiency improvement strategy corresponding to the operator information to update and set each operator to obtain the target operator.

[0136] Loading model parameters: The Graph Executor calls the optimized target operator to sequentially execute each operator node in the model's internal representation, building the target task model. Furthermore, the Distributed Scheduler can be used to distribute the execution of each operator node in the model's internal representation, improving the efficiency of model acceleration optimization.

[0137] It should be noted that after the graph executor obtains the target task model, it can process the input task (input tensor) and output the corresponding task processing result (output tensor). At the same time, users can also configure the .yml config file to adjust the model functions according to their needs.

[0138] Specifically, as shown in FIG4 , the following steps S41 to S45 can be performed to optimize the computational efficiency of the operators of the model units in the initial task model using an optimization strategy to obtain the target task model:

[0139] Step S41: parse the initial task model to obtain operators of multiple model units in the initial task model.

[0140] Specifically, the initial task model refers to a model with task processing capabilities, such as a recurrent neural network (RNN), a convolutional neural network (CNN), a pre-trained large model, a multimodal pre-trained model, etc., which is selected according to the actual situation and is not limited in this embodiment. The multiple model units in the initial task model can also be understood as the various processing layers in the initial task model, such as a normalization layer, an attention layer, etc.

[0141] Step S42: Using each operator as a node, construct a model representation graph corresponding to the initial task model.

[0142] Step S43: identifying operator information of each operator in the model representation graph.

[0143] Step S44: Utilize the computing efficiency improvement strategy corresponding to the operator information to optimize the computing efficiency of each operator and obtain the target operator.

[0144] In a first optional implementation of step S44, the first operators of the target model unit can be fused to construct a target operator, where the first operator of the target model unit is an arithmetic operation type and the target model unit is any model unit in the initial task model. This can fully utilize the continuity of CPU and GPU memory read and write, significantly reduce operator input and output overhead, and save some redundant calculations.

[0145] For example, the element-wise multiplication (MatMul) and element-wise addition (BiasAdd) operators can be fused into a single GEMM operation, multiple transposition (Transpose), multiplication (BatchMatMul), and multiplication (Mul) operators can be fused into a single BatchGEMM operation, and the addition (Add) and normalization (Softmax) operators can be fused into a single masked normalization (MaskedSoftmax) operation. It should be noted that the server has comprehensively updated and optimized the GEMM operator. Specific measures include, but are not limited to, using FP16 and BF16 data formats to increase throughput, using INT8 for dynamic quantization, using the NC4HW4 data layout format, fully utilizing the concurrent read and write capabilities of vector registers, using a parallel computing framework, hand-writing the GEMM kernel for the CPU version, and fully utilizing the L1 cache (first-level cache), L2 cache (second-level cache), vector registers, and concurrent pipelines. The specific selection is based on actual conditions and is not limited in this embodiment.

[0146] In a second optional implementation of step S44, the operator information includes a transposition operation type. The server may obtain a preset stride parameter and construct a target operator based on the stride parameter, wherein the target operator is configured to store target feature data in memory according to the stride parameter and read the target feature data from the memory when performing calculations on the target feature data, the target feature data being the feature data subjected to the transposition operation.

[0147] Specifically, the stride parameter is a pre-set one-dimensional vector with a length of 4. The format is [a, x, y, z], which represents [batch sliding step, horizontal sliding step, vertical sliding step, channel sliding step] respectively. The general form of the stride parameter is [1, x, y, 1]. The first 1 means: the sliding step in the batch dimension is 1, that is, no sample is skipped. x represents: the horizontal sliding step of the convolution kernel. y represents: the vertical sliding step of the convolution kernel. The last 1 represents: the sliding step in the channel dimension is 1, that is, no color channel is skipped. Among them, the stride parameter is pre-set according to the matrix calculated by GEMM. Batch represents the number of data groups.

[0148] For example, in the calculation of attention, it is necessary to deform (Reshape) the two-dimensional matrix into a three-dimensional matrix, and then transpose (Transpose) it into a locally continuous memory arrangement, and then perform matrix multiplication calculations. In this optional implementation, the stride parameter can be added to implement the stride_GEMM operator. Since the traditional GEMM can only access memory sequentially, the original matrix must be deformed and reorganized through Reshape+Transpose before it can be calculated. The stride_GEMM can jump to the specified memory for calculation by providing the corresponding stride parameter, realizing a GEMM that can jump memory access, allowing the original matrix to be calculated directly, thereby saving the overhead of the two operators Reshape and Transpose, and improving the model operation efficiency. The Reshape operator is used to reset the data arrangement form of the elements.

[0149] In the third optional implementation of step S44, a pre-set target matrix can be obtained when the operator is in a specified form, wherein the number of non-zero elements in each column of the target matrix is ​​the same; based on the target matrix, a target operator is constructed. Since common scientific computing is not designed for sparse GEMM, the calculation mode is generally dense(A)*sparse(B)=dense(C), so many transpositions need to be done in the implementation before it can be used. Therefore, in this optional implementation, the server includes a sparse format and its corresponding sparse kernel, supports the calculation method of sparse(A)*dense(B)=dense(C), and avoids the additional overhead caused by the transposition operation. Among them, A, B, and C are matrices in matrix multiplication, representing that the A matrix is ​​multiplied by the B matrix to obtain the C matrix.

[0150] For example, the server aligns memory addresses using pad (data padding characters) for traditional column-major order (CSC) and row-major order (CSR), uses efficient memory access instructions, and each warp (platform resource scheduling unit) calculates one element to ensure continuous memory access. It also uses the global cache to cache each row of the A matrix and the L1 cache (first-level cache) to cache each column of the B matrix (CSR / CSC format), fully utilizing the CPU's cache capacity. In addition, the server has customized an efficient sparse format ELL for some specific model structures. This format is suitable for operators of a specified form, where the specified form means that the number of non-zero elements in each column of the matrix is ​​the same and the distribution of non-zero elements in each column is completely random. Since the target matrix B has the same number of non-zero elements in each column, the problem of load imbalance is avoided. The global cache is used to cache each row of the A matrix, and registers are used to cache each column of the B matrix, fully utilizing the cache capacity under non-limited input scales while improving parallelism. Each thread simultaneously calculates one element, performing thread accumulation. Each warp calculates 32 adjacent elements, ensuring continuous memory access while avoiding intra-warp reduction overhead. Furthermore, the server framework automatically selects matrices based on performance, avoiding load imbalance and reducing the overhead associated with transposition operations.

[0151] In a fourth optional implementation of this step, the operator information includes a normalization type, and the server obtains a preset out-of-order parameter; a target operator is constructed based on the out-of-order parameter, wherein the target operator is used to perform parallel reduction on the normalized feature data according to the out-of-order parameter, and to perform operations on the normalized feature data after parallel reduction, which can reduce overhead and improve model operation efficiency.

[0152] Exemplarily, before data reduction, the operators performing operations on the normalized feature data include averaging (ReduceMean), subtraction (Sub), power (Pow), addition (Add), square root (Sqrt), division (Div), and multiplication (Mul). In this embodiment, after data reduction, for the layer normalization (LayerNorm) operator, the server uses a kernel to directly implement it: small kernel fusion reduces the overhead of launching the kernel, optimizes the reduce method, uses shared memory and warp-level shuffle source language to accelerate the reduce calculation, that is, constructs the target operator based on the out-of-order parameter (shfl_down_sync). Among them, reduce is a form of a continuous segment of the same operation such as accumulation or multiplication. Data in the same warp can be exchanged at high speed through shuffle instructions.

[0153] In a fifth optional implementation of this step, the operator information includes the activation type. The server post-places the activation type operator into the execution of the target operator and then updates the execution logic of the target operator. This can reduce kernel launch overhead while achieving a balance between accuracy and speed.

[0154] For example, multiple methods can be used to update the GELU operator. For example, kernel fusion can be used to execute activation operators and GEMMs serially. This means that activation operators such as GELU and RELU are directly post-processed into the GEMM calculation process, allowing the activation operators and GEMM to be calculated simultaneously. It should be noted that the specific activation operator type can be selected based on the downstream tasks of the model.

[0155] In a sixth optional implementation of this step, the operator information includes a decoding type. The server obtains a decoding result corresponding to the operator of the decoding type; stores the decoding result in a pre-set decoding cache unit; and updates the decoding logic of the operator based on the decoding result in the decoding cache unit to obtain a target operator. This avoids unnecessary repeated calculations during the decoding process and improves the model operation speed.

[0156] For example, before the decoding operator is optimized, the decoding process involves a lot of repetitive computation. Each decoding operation requires recalculating the vector representation of the previously generated token (character). For example, when generating ID1 in the first step, generating ID2 from ID1 requires recalculating ID1 again in the second step. And when generating ID3 from ID2 in the third step, generating ID1 and ID2 again requires recalculating ID3. Therefore, a decoding cache can be added to the server to record the decoding results of each step, eliminating unnecessary recalculation.

[0157] In a seventh optional implementation of this step, the server may store the weights corresponding to the target operators according to a specified weight storage format, where the specified weight storage format includes operator attribute information and model execution data. Specifically, the specified weight storage format is a binary weight storage format. Operator attribute information includes basic information such as weight name, data type, and shape, as well as special information such as sparsity type and segmentation type. Model execution data refers to the specific numerical values ​​of the weights.

[0158] For example, the weights and model representation graph corresponding to the target operator can be stored in the same .as file. In another possible implementation of this specification, the weights and model representation graph corresponding to the target operator can be stored separately, that is, the .as file is used to store the model representation graph structure, and the .asparam file is used to store the weights corresponding to the target operator. When loading a very large-scale model, separate storage is generally used to prevent the memory from being overwhelmed by loading a large amount of temporarily useless weight data during the model structure parsing stage.

[0159] Step S45: Construct a target task model based on the target operator.

[0160] In this step, the weight of the target operator is divided according to multiple operator running units to obtain the corresponding dividing weight of each operator running unit; the execution results obtained by each operator running unit executing the target operator according to the dividing weight are received; the execution results of each operator running unit are merged to obtain the target task model.

[0161] For example, in one implementation of operator weight splitting, the model performance can be further optimized by Attention operator fusion based on model parallelism. Originally, the weight on each card was obtained by splitting weightQ, weightK, and weightV respectively. In the embodiment of this specification, these three weights can be fused into weightQKV, and the splitting of the three parts of the weight can be carried out simultaneously, which reduces the operator's input and output overhead and avoids some redundant calculations. Specifically, the weight before fusion is the memory block corresponding to Q, K, and V in the upper half of the figure. For example, Q corresponds to Q1+Q2, and it will not be separated in a single-card scenario. In a multi-card scenario, a weightQKV needs to be assigned to each card. (Q1, K1, V1) can be extracted as the weightsQKV of the first graphics card, and so on. (Q2, K2, V2) can be extracted as the weightsQKV of the second graphics card. Among them, weightQ, weightK, and weightV are the three weight names in the model.

[0162] It's important to note that the model sharding process is performed internally by the Distributed Scheduler. Simply providing the original weight file and specifying the number of GPUs at runtime automatically completes the sharding process, eliminating the need for user-defined sharding. This prevents the sharded weights from only running on a fixed number of GPUs. The sharded data is saved as a binary file in an internal format that can be processed by microcomputers. Subsequent runs of the same model can directly read the data without re-sharding, reducing model service startup time.

[0163] For example, in an example of an operator weight splitting operation, the input is calculated by GEMM twice, assuming that there are X graphics cards calculating in parallel. By cutting weight1 in the first GEMM into X parts vertically, each graphics card will get a vertical result inner_i, and the actual result inner can be obtained after merging. Then the weight2 of the second GEMM is cut into X parts horizontally, and a horizontal result weight2_i is obtained. At this time, executing GEMM (inner_i, weight2_i) on each graphics card will get an output component out_i of the same size as the actual output matrix. Finally, the all reduce_sum operation is used to add up all out_i to get the final output result output. Among them, weight1 and weight2 represent the weights of the two GEMM calculations respectively.

[0164] For example, in another example of operator weight splitting, the input undergoes two GEMM calculations, assuming that there are X graphics cards performing parallel calculations. By cutting weight1 in the first GEMM into X parts vertically, each graphics card will obtain a vertical result inner_i, which can be merged to obtain the actual result inner. Then, weight2 of the second GEMM is cut into X parts vertically, and a horizontal result weight2_i is obtained. At this time, executing GEMM(inner_i, weight2_i) on each graphics card will obtain an out_i of the same size as the actual output matrix. Finally, all out_i are merged using the all gather operation to obtain the final result output.

[0165] By applying the solution of the embodiments of this specification, the single-machine multi-card mode supports the simultaneous use of multiple graphics cards on a single device for model inference, solving the problem of insufficient video memory on a single graphics card, and can utilize weight splitting strategies and the resources of multiple graphics cards to achieve performance acceleration.

[0166] The solution of this embodiment parses the initial task model to obtain the operators of multiple model units in the initial task model; constructs a model representation graph corresponding to the initial task model with each operator as a node; identifies the operator information of each operator in the model representation graph; uses the computational efficiency improvement strategy corresponding to the operator information to optimize the computational efficiency of each operator to obtain the target operator; and constructs the target task model based on the target operator. Since the target task model is obtained by optimizing the computational efficiency of the operators of each model unit based on the operator information of each model unit in the initial task model using the corresponding computational efficiency improvement strategy, the computational efficiency of the task model can be improved while maintaining the processing accuracy of the task model, greatly shortening the task processing time and improving the efficiency of task processing.

[0167] Exemplarily, FIG5 is an architectural diagram of the multimodal task processing method provided by the present application. As shown in FIG5 , taking the input multimodal input data including a picture and the text “Who is the person in the picture?” as an example, the picture is encoded as a feature representation of the picture, and the picture in the multimodal input data is replaced with the corresponding modal identification information 10001 to obtain a plain text input. The plain text input is encoded to obtain a feature representation of the plain text input. Furthermore, by replacing the feature representation of the picture, multimodal fusion is achieved to obtain a feature representation of multimodal fusion. Furthermore, efficient reasoning is performed based on the feature representation of multimodal fusion to obtain the task processing result.

[0168] The present application provides a multimodal task processing system, including a server and an end-side device. FIG6 is an interaction flow chart of a multimodal task processing system provided by an exemplary embodiment of the present application. As shown in FIG6, the interaction flow between the server and the end-side device in the multimodal task processing system is as follows:

[0169] Step S61: The terminal-side device receives multimodal input data to be processed input by the user.

[0170] The multimodal input data input by the user in this step is similar to the content of the multimodal input data obtained by the server in the aforementioned step S21. For details, please refer to the relevant content of the aforementioned embodiment and will not be repeated here.

[0171] Step S62: The client device sends multimodal input data to the server.

[0172] Step S63: The server receives multimodal input data sent by the terminal-side device.

[0173] Step S64: The server inputs the multimodal input data into the target task model, encodes and fuses the multimodal input data through the target task model to obtain a feature representation of the multimodal fusion, and performs task processing based on the feature representation of the multimodal fusion to generate a task processing result; wherein, the target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy.

[0174] The specific implementation method of this step is consistent with the implementation method of the aforementioned step S22. Please refer to the relevant content of the aforementioned embodiment for details, and no further details will be given here.

[0175] Step S65: The server sends the task processing result to the terminal device.

[0176] Step S66: The terminal-side device receives the task processing result and outputs the task processing result.

[0177] This embodiment provides an interactive flowchart of a multimodal task processing system, which can achieve efficient multimodal task processing and improve the reasoning efficiency and performance of multimodal tasks.

[0178] Figure 7 is a flowchart of a multimodal dialogue task processing method provided by an exemplary embodiment of the present application, which is applied to a server in a human-computer dialogue scenario. As shown in Figure 7, the specific steps of the method are as follows:

[0179] Step S71: A dialogue task request sent by a receiving end-side device is received, where the dialogue request includes multimodal input data.

[0180] This embodiment is applied to a human-computer dialogue scenario. A user interacts with a server through a client device and sends a dialogue task request containing multimodal input data to the server. The server retrieves the multimodal input data from the received dialogue task request. The server generates a response text based on the multimodal input data and sends it to the client device.

[0181] The multimodal input data included in the dialogue task request is multimodal data input and / or specified by the user through the terminal device, which is similar to the content of the multimodal input data obtained by the server in the aforementioned step S21. For details, please refer to the relevant content of the aforementioned embodiment and will not be repeated here.

[0182] Step S72: input the multimodal input data into the target dialogue macromodel, encode and fuse the multimodal input data through the target dialogue macromodel to obtain a feature representation of the multimodal fusion, and perform dialogue task processing based on the feature representation of the multimodal fusion to generate a reply text; wherein, the target dialogue macromodel is obtained by optimizing the computational efficiency of the operators of the model units in the initial dialogue macromodel using an optimization strategy.

[0183] Among them, the initial dialogue large model can adopt a large model that realizes the human-computer dialogue function, such as a pre-trained language large model, a multimodal pre-trained model, etc., which is not specifically limited here.

[0184] The specific implementation method of this step is consistent with the implementation method of the aforementioned step S22. Please refer to the relevant content of the aforementioned embodiment for details, and no further details will be given here.

[0185] Step S73: Send a reply text to the end-side device.

[0186] In this embodiment, the target conversational model is obtained by optimizing the computational efficiency of the operators of the model units in the initial conversational model using an optimization strategy. Specifically, the target conversational model can be obtained by updating and configuring the operators of each model unit in the initial conversational model using the corresponding computational efficiency improvement strategy based on the operator information of each model unit in the initial conversational model. Because the target conversational model is obtained by updating and configuring the operators of each model unit using the corresponding computational efficiency improvement strategy based on the operator information of each model unit in the initial conversational model, the computational efficiency of the conversational model can be improved while maintaining the inference accuracy of the conversational model. This significantly reduces the processing time of multimodal conversational tasks and improves the efficiency and performance of multimodal conversational task processing.

[0187] In an optional embodiment, in step S72, the multimodal input data is encoded and fused by the target dialogue macro model to obtain a multimodal fused feature representation, which can be specifically implemented in the following manner:

[0188] The non-text modal inputs contained in the multimodal input data are encoded separately to obtain feature representations of the non-text modal inputs; the non-text modal inputs in the multimodal input data are replaced with corresponding modal identification information to obtain plain text inputs; the plain text inputs are encoded to obtain feature representations of the plain text inputs; the feature representations of the non-text modal inputs are used to replace the feature representations of the corresponding modal identification information in the feature representations of the plain text inputs to obtain feature representations of multimodal fusion.

[0189] This implementation method is consistent with the implementation method of the aforementioned steps S31-S34. Please refer to the relevant content of the aforementioned embodiment for details, and this embodiment will not be repeated here.

[0190] The present application provides a multimodal conversation task processing system, including a server and a terminal device. FIG8 is an interaction flow chart of the multimodal conversation task processing system provided by an exemplary embodiment of the present application. As shown in FIG8, the interaction flow between the server and the terminal device in the multimodal conversation task processing system is as follows:

[0191] Step S81: The end-side device receives a dialogue task request sent by the end-side device, where the dialogue task request includes multimodal input data.

[0192] The multimodal input data input by the user in this step is similar to the content of the multimodal input data obtained by the server in the aforementioned step S71. For details, please refer to the relevant content of the aforementioned embodiment and will not be repeated here.

[0193] Step S82: The client device sends multimodal input data to the server.

[0194] Step S83: The server receives multimodal input data.

[0195] Step S84: The server inputs the multimodal input data into the target dialogue macromodel, encodes and fuses the multimodal input data through the target dialogue macromodel to obtain a feature representation of the multimodal fusion, and performs dialogue task processing based on the feature representation of the multimodal fusion to generate a reply text; wherein, the target dialogue macromodel is obtained by optimizing the computational efficiency of the operators of the model units in the initial dialogue macromodel using an optimization strategy.

[0196] The specific implementation method of this step is consistent with the implementation method of the aforementioned step S22. Please refer to the relevant content of the aforementioned embodiment for details, and no further details will be given here.

[0197] Step S85: The server is also used to send a reply text to the end-side device.

[0198] Step S86: The terminal-side device is also used to receive and output the reply text.

[0199] This embodiment provides an interaction flow chart of a multimodal dialogue task processing system, which can efficiently implement the processing of multimodal dialogue tasks and improve the reasoning efficiency and performance of multimodal dialogue tasks.

[0200] Figure 9 is a schematic diagram of the structure of a multimodal task processing device provided in an exemplary embodiment of the present application. The multimodal task processing device provided in this embodiment can execute the server processing flow in the exemplary embodiment of the multimodal task processing method. As shown in Figure 9, the multimodal task processing device 90 includes a multimodal data acquisition module 91, a multimodal task processing module 92, and a model optimization module 93.

[0201] The multimodal data acquisition module 91 is used to acquire multimodal input data to be processed.

[0202] The multimodal task processing module 92 is used to input multimodal input data into the target task model, encode and fuse the multimodal input data using the target task model to obtain a multimodal fusion feature representation, and then perform task processing based on the multimodal fusion feature representation to generate a task processing result. The target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy.

[0203] The model optimization module 93 is used to optimize the computational efficiency of the operators of the model units in the initial task model using an optimization strategy to obtain a target task model.

[0204] In an optional embodiment, when encoding and fusing multimodal input data to obtain a multimodal fused feature representation, the multimodal task processing module 92 is further configured to:

[0205] The non-text modal inputs contained in the multimodal input data are encoded separately to obtain feature representations of the non-text modal inputs; the non-text modal inputs in the multimodal input data are replaced with corresponding modal identification information to obtain plain text inputs; the plain text inputs are encoded to obtain feature representations of the plain text inputs; the feature representations of the non-text modal inputs are used to replace the feature representations of the corresponding modal identification information in the feature representations of the plain text inputs to obtain feature representations of multimodal fusion.

[0206] In an optional embodiment, after respectively encoding the non-text modality inputs included in the multimodal input data to obtain feature representations of the non-text modality inputs, the multimodal task processing module 92 is further configured to:

[0207] A sequence consisting of feature representations of at least one input of the same non-text modality in the multimodal input data is stored in correspondence with modality identification information corresponding to the non-text modality.

[0208] In an optional embodiment, when replacing non-text modal input in multimodal input data with corresponding modality identification information to obtain plain text input, the multimodal task processing module 92 is further configured to:

[0209] The non-text modal input in the multimodal input data is replaced by a text segment of the same length spliced ​​using the corresponding modal identification information according to the length of the non-text modal input to obtain a plain text input.

[0210] In an optional embodiment, when using the feature representation of non-text modal input to replace the feature representation of the modality identification information in the feature representation of pure text input to obtain a multimodal fusion feature representation, the multimodal task processing module 92 is further configured to:

[0211] For the feature representation of any non-text modal input, based on the length of the feature representation of the non-text modal input, search for a text fragment of the same length spliced ​​together with the corresponding modal identification information in the plain text input to determine the position of the text fragment in the plain text input; based on the position of the text fragment in the plain text input, use the feature representation of the non-text modal input to replace the content at the corresponding position in the feature representation of the plain text input to obtain a multimodal fused feature representation.

[0212] In an optional embodiment, the multimodal input data to be processed includes multimodal input data of multiple task requests, and the multiple task requests constitute a group of requests. When encoding the plain text input to obtain the feature representation of the plain text input, the multimodal task processing module 92 is further configured to:

[0213] The plain text input of each task request is passed into the preset encoding interface. The maximum length of the plain text input of each task request is set through the preset encoding interface. By filling a placeholder at the end of the plain text input whose length is less than the maximum length, the lengths of the plain text inputs of multiple task requests are aligned, and the aligned plain text inputs of the multiple task requests are encoded.

[0214] Furthermore, when encoding the aligned plain text inputs of multiple task requests, the multimodal task processing module 92 is further configured to:

[0215] Parse the placeholders contained in the aligned plain text input to determine the actual effective length of the aligned plain text input and the placeholder offsets of the placeholders contained therein; and encode the actual effective inputs in the aligned plain text inputs of multiple task requests based on the actual effective length of the aligned plain text input and the placeholder offsets of the placeholders contained therein.

[0216] In an optional embodiment, the model optimization module 93 is specifically used to: before inputting the multimodal input data into the target task model, parse the initial task model to obtain the operators of multiple model units in the initial task model; construct a model representation diagram corresponding to the initial task model with each operator as a node; identify the operator information of each operator in the model representation diagram; use the computational efficiency improvement strategy corresponding to the operator information to optimize the computational efficiency of each operator to obtain the target operator; and construct the target task model based on the target operator.

[0217] The device provided in the embodiment of the present application can be specifically used to execute the processing flow executed by the server in any of the above method embodiments, and the specific functions implemented and the technical effects that can be achieved will not be repeated here.

[0218] Figure 10 is a schematic diagram of the structure of a server provided in an embodiment of the present application. As shown in Figure 10, the server includes: a memory 1001 and a processor 1002. The memory 1001 is used to store computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 1002 is communicatively connected to the memory 1001 and is used to execute the computer-executable instructions stored in the memory 1001 to implement the technical solution provided by any of the above-mentioned method embodiments. Its specific functions and the technical effects that can be achieved are similar and will not be repeated here. Figure 10 takes the server as a cloud server deployed in the cloud as an example for schematic illustration. The server can also be a server deployed locally, which is not specifically limited here.

[0219] Optionally, as shown in Figure 10, the server further includes other components such as a firewall 1003, a load balancer 1004, a communication component 1005, and a power supply component 1006. Figure 10 only schematically illustrates some components, which does not mean that the server only includes the components shown in Figure 10.

[0220] An embodiment of the present application also provides a computer-readable storage medium, which stores computer execution instructions. When the computer execution instructions are executed by a processor, they are used to implement the technical solutions provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0221] An embodiment of the present application also provides a computer program product, which includes: a computer program, the computer program is stored in a readable storage medium, at least one processor of the server can read the computer program from the readable storage medium, and at least one processor executes the computer program so that the server executes the technical solution provided by any of the above method embodiments. The specific functions and technical effects that can be achieved are not repeated here.

[0222] The present application provides a chip comprising: a processing module and a communication interface. The processing module is capable of executing the technical solution of the server in the aforementioned method embodiments. Optionally, the chip further comprises a storage module (e.g., a memory) configured to store instructions, and the processing module configured to execute the instructions stored in the storage module. Execution of the instructions stored in the storage module causes the processing module to execute the technical solution provided by any of the aforementioned method embodiments.

[0223] The aforementioned storage may be an object storage service (OSS). The aforementioned storage may be implemented by any type of volatile or non-volatile storage device, or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0224] The communication component is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on communication standards, such as mobile hotspot (WiFi), second-generation mobile communication system (2G), third-generation mobile communication system (3G), fourth-generation mobile communication system (4G) / long-term evolution (LTE), fifth-generation mobile communication system (5G), or a combination thereof. In an exemplary embodiment, the communication component receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies. The power supply component provides power to various components of the device where the power supply component is located. The power supply component may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device where the power supply component is located.

[0225] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0226] The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device produce a device for implementing the functions specified in one or more processes in the flowchart and / or one or more boxes in the block diagram.

[0227] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, such that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device that implements the functions specified in one or more processes of the flowchart and / or one or more blocks of the block diagram. These computer program instructions may also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable device provide steps for implementing the functions specified in one or more processes of the flowchart and / or one or more blocks of the block diagram.

[0228] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory. Memory may include non-permanent storage in a computer-readable medium, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0229] Computer-readable media includes permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include transitory computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0230] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0231] In addition, some of the processes described in the above embodiments and the accompanying drawings include multiple operations that appear in a specific order, but it should be clearly understood that these operations may not be executed in the order in which they appear in this document or in parallel. They are only used to distinguish between different operations, and the sequence number itself does not represent any order of execution. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequence, nor do they limit "first" and "second" to different types. The meaning of "multiple" is more than two, unless otherwise clearly and specifically defined.

[0232] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of the present application and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0233] It should be understood that the present application is not limited to the exact structure described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A multimodal task processing method, characterized in that: include: Obtain multimodal input data to be processed; The multimodal input data is input into a target task model, the multimodal input data is encoded and fused through the target task model to obtain a feature representation of multimodal fusion, and task processing is performed according to the feature representation of multimodal fusion to generate a task processing result; wherein the target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy.

2. The method according to claim 1, characterized in that The encoding and fusing of the multimodal input data to obtain a multimodal fused feature representation includes: Encoding the non-text modality inputs contained in the multimodal input data respectively to obtain feature representations of the non-text modality inputs; Replacing the non-text modality input in the multimodal input data with the corresponding modality identification information to obtain a plain text input; Encoding the plain text input to obtain a feature representation of the plain text input; The feature representation of the non-text modal input is used to replace the feature representation of the modality identification information in the feature representation of the plain text input, so as to obtain a multi-modal fused feature representation.

3. The method according to claim 2, characterized in that After respectively encoding the non-text modal inputs contained in the multimodal input data to obtain feature representations of the non-text modal inputs, the method further includes: A sequence consisting of feature representations of at least one input of the same non-text modality in the multimodal input data is stored in correspondence with modality identification information corresponding to the non-text modality.

4. The method according to claim 2, characterized in that: The step of replacing the non-text modal input in the multimodal input data with corresponding modal identification information to obtain a plain text input includes: The non-text modal input in the multimodal input data is replaced with a text segment of the same length spliced ​​together using corresponding modal identification information according to the length of the non-text modal input to obtain a plain text input.

5. The method according to claim 4, characterized in that The step of using the feature representation of the non-text modal input to replace the feature representation of the modal identification information in the feature representation of the plain text input to obtain a multi-modal fusion feature representation includes: For any feature representation of the non-text modal input, according to the length of the feature representation of the non-text modal input, searching in the plain text input for a text segment of the same length spliced ​​with corresponding modal identification information, and determining the position of the text segment in the plain text input; According to the position of the text fragment in the plain text input, the feature representation of the non-text modal input is used to replace the content at the corresponding position in the feature representation of the plain text input to obtain a multimodal fused feature representation.

6. The method according to any one of claims 2 to 5, characterized in that: The multimodal input data to be processed includes multimodal input data of a plurality of task requests, wherein the plurality of task requests constitute a group of requests; The step of encoding the plain text input to obtain a feature representation of the plain text input includes: The plain text input of each task request is passed into a preset encoding interface, and the maximum length of the plain text input of each task request is checked through the preset encoding interface. By filling a placeholder at the end of the plain text input whose length is less than the maximum length, the lengths of the plain text inputs of the multiple task requests are aligned, and the aligned plain text inputs of the multiple task requests are encoded.

7. The method according to claim 6, characterized in that The encoding of the aligned plain text inputs of the plurality of task requests comprises: Parsing the placeholders contained in the aligned plain text input to determine the actual effective length of the aligned plain text input and the placeholder offsets of the contained placeholders; According to the actual effective length of the aligned plain text input and the placeholder offset of the included placeholder, the actual effective input in the aligned plain text input of multiple task requests is encoded.

8. The method according to any one of claims 1 to 5, characterized in that Before inputting the multimodal input data into the target task model, the method further includes: Parsing the initial task model to obtain operators of multiple model units in the initial task model; Taking each operator as a node, constructing a model representation graph corresponding to the initial task model; Identifying operator information of each operator in the model representation graph; The computing efficiency improvement strategy corresponding to the operator information is used to improve the computing efficiency of each operator. Optimize settings to obtain the target operator; According to the target operator, a target task model is constructed.

9. A multimodal dialogue task processing method, characterized in that: Applicable to servers, including: A dialog task request sent by a receiving end-side device, wherein the dialog request includes multimodal input data; Inputting the multimodal input data into the target dialogue macromodel, encoding and fusing the multimodal input data through the target dialogue macromodel to obtain a multimodal fused feature representation, and performing dialogue task processing according to the multimodal fused feature representation to generate a reply text; wherein the target dialogue macromodel is obtained by optimizing the computational efficiency of the operators of the model units in the initial dialogue macromodel using an optimization strategy; The reply text is sent to the end-side device.

10. The method according to claim 9, characterized in that The encoding and fusing of the multimodal input data to obtain a multimodal fused feature representation includes: Encoding the non-text modality inputs contained in the multimodal input data respectively to obtain feature representations of the non-text modality inputs; Replacing the non-text modality input in the multimodal input data with the corresponding modality identification information to obtain a plain text input; Encoding the plain text input to obtain a feature representation of the plain text input; The feature representation of the non-text modal input is used to replace the feature representation of the modality identification information in the feature representation of the plain text input, so as to obtain a multi-modal fused feature representation.

11. A multimodal dialogue task processing system, characterized in that: Including end-side devices and servers, The terminal side device is used to receive a dialogue task request sent by the terminal side device, and send multimodal input data to the server, wherein the dialogue task request includes the multimodal input data; The server is used to receive multimodal input data, input the multimodal input data into a target dialogue macromodel, encode and fuse the multimodal input data through the target dialogue macromodel to obtain a multimodal fused feature representation, and perform dialogue task processing according to the multimodal fused feature representation to generate a reply text; wherein the target dialogue macromodel is obtained by optimizing the computational efficiency of operators of model units in the initial dialogue macromodel using an optimization strategy; The server is also used to send the reply text to the terminal device; The terminal side device is also used to receive and output the reply text.

12. A multimodal task processing system, characterized in that: Including end-side devices and servers, The terminal side device is used to receive multimodal input data to be processed input by a user, and send the multimodal input data to the server; The server is used to receive multimodal input data sent by the terminal device, input the multimodal input data into the target task model, encode and fuse the multimodal input data through the target task model to obtain a feature representation of multimodal fusion, and perform task processing according to the feature representation of multimodal fusion to generate a task processing result; wherein the target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy; The server is also used to send the task processing result to the terminal device; The terminal side device is also used to receive the task processing result and output the task processing result.

13. A server, characterized in that: include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory to implement the method according to any one of claims 1 to 10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 10 when executed by a processor.

Images