Method, system and apparatus for multi-modal task processing and dialog task processing

Abstract

The application provides a multi-modal task processing and dialogue task processing method, system and device. The method of the application can be applied to a large model-based task processing scene, and the operator of the model unit in the initial task model is optimized to obtain a target task model through an optimization strategy, so as to improve the operation efficiency of the large model. In multi-modal task processing, the multi-modal input data is input into the target task model, the multi-modal input data is encoded and fused through the target task model to obtain multi-modal fusion feature representation, and task processing is performed according to the multi-modal fusion feature representation to generate a task processing result, thereby realizing an inference mode capable of accepting multi-modal input. Based on the target task model obtained through operation efficiency optimization, the operation efficiency of the model is improved under the premise of maintaining the model accuracy, the multi-modal task processing time is greatly shortened, and the efficiency and performance of the multi-modal task processing are improved.

Description

Technical Field

[0001] This application relates to computer technology, and more particularly to a method, system, and device for multimodal task processing and dialogue task processing. Background Technology

[0002] With the rise and widespread application of Natural Language Processing (NLP), a large number of Natural Language Generation (NLG) models have emerged in the industry. Through these models, an increasing number of work and learning tasks can be automated.

[0003] Traditional Natural Language Generator (NLG) models typically use text as both input and output, and their encoding modules only support text encoding. However, with advancements in technology and technology, simple text is no longer sufficient to demonstrate a model's capabilities. More and more models are using multimodal data such as images, audio, and video for training and inference, ultimately enabling them to understand multimodal data and perform multimodal tasks such as "image description" and "summarizing video content."

[0004] However, such models still have performance issues when performing multimodal inference. Multimodal data takes up a lot of input length. For example, an image, after being encoded, often takes up the equivalent of hundreds of characters in input length, which greatly affects the overall inference performance of the model, resulting in long inference time and low performance for multimodal tasks. Summary of the Invention

[0005] This application provides a method, system, and device for multimodal task processing and dialogue task processing, in order to solve the problems of long inference time and low performance in multimodal tasks.

[0006] Firstly, this application provides a multimodal task processing method, including:

[0007] Acquire multimodal input data to be processed; input the multimodal input data into the target task model, encode and fuse the multimodal input data through the target task model to obtain a multimodal fused feature representation, and perform task processing according to the multimodal fused feature representation to generate task processing results; wherein, the target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy.

[0008] Secondly, this application provides a multimodal dialogue task processing method, applied to a server, comprising:

[0009] The system receives a dialogue task request sent by a receiving end device, the dialogue request containing multimodal input data; inputs the multimodal input data into a target dialogue model, encodes and fuses the multimodal input data through the target dialogue model to obtain a multimodal fused feature representation, and performs dialogue task processing based on the multimodal fused feature representation to generate a response text; wherein, the target dialogue model is obtained by optimizing the computational efficiency of the operators of the model units in the initial dialogue model using an optimization strategy; and sends the response text to the receiving end device.

[0010] Thirdly, this application provides a multimodal dialogue task processing system, including an end-side device and a server.

[0011] The endpoint device is used to receive a dialogue task request sent by the endpoint device and send multimodal input data to the server. The dialogue task request includes multimodal input data.

[0012] The server is used to receive multimodal input data, input the multimodal input data into a target dialogue model, encode and fuse the multimodal input data through the target dialogue model to obtain a multimodal fused feature representation, and perform dialogue task processing based on the multimodal fused feature representation to generate response text; wherein, the target dialogue model is obtained by optimizing the computational efficiency of the operators of the model units in the initial dialogue model using an optimization strategy;

[0013] The server is also used to send the response text to the end-side device;

[0014] The end-side device is also used to receive and output the response text.

[0015] Fourthly, this application provides a multimodal task processing system, including an edge device and a server.

[0016] The end-side device is used to receive multimodal input data to be processed from the user and send the multimodal input data to the server;

[0017] The server is used to receive multimodal input data sent by the end-side device, input the multimodal input data into the target task model, encode and fuse the multimodal input data through the target task model to obtain a multimodal fused feature representation, and perform task processing based on the multimodal fused feature representation to generate task processing results; wherein, the target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy;

[0018] The server is also used to send the task processing result to the end-side device;

[0019] The end-side device is also used to receive task processing results and output the task processing results.

[0020] Fifthly, this application provides a server, including: a processor, and a memory communicatively connected to the processor; the memory stores computer-executable instructions; the processor executes the computer-executable instructions stored in the memory to implement the method as described in the first or second aspect.

[0021] In a sixth aspect, this application provides a computer-readable storage medium storing computer-executable instructions that, when executed by a processor, are used to implement the method described in the first or second aspect.

[0022] The multimodal task processing and dialogue task processing methods, systems, and devices provided in this application obtain a target task model by optimizing the computational efficiency of operators in the model units of the initial task model using an optimization strategy. During multimodal task processing, multimodal input data to be processed is acquired and input into the target task model. The target task model encodes and fuses the multimodal input data to obtain a multimodal fused feature representation. Task processing is then performed based on this multimodal fused feature representation to generate task processing results. This achieves a reasoning mode capable of accepting multimodal input. Furthermore, since the target task model used for multimodal task processing is obtained by optimizing the computational efficiency of operators in the model units of the initial task model using an optimization strategy, the computational efficiency of the model can be improved while maintaining model accuracy without degradation, significantly shortening the multimodal task processing time and improving the efficiency and performance of multimodal task processing. Attached Figure Description

[0023] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0024] Figure 1 A schematic diagram of an example task processing system architecture to which this application applies;

[0025] Figure 2 A flowchart of a multimodal task processing method provided in an exemplary embodiment of this application;

[0026] Figure 3 A flowchart of a multimodal coding fusion method provided for an exemplary embodiment of this application;

[0027] Figure 4 A flowchart illustrating a method for optimizing computational efficiency of a model, provided as an exemplary embodiment of this application;

[0028] Figure 5 An architecture diagram of a multimodal task processing method provided for an exemplary embodiment of this application;

[0029] Figure 6 An interactive flowchart of a multimodal task processing system provided as an exemplary embodiment of this application;

[0030] Figure 7 A flowchart of a multimodal dialogue task processing method provided for an exemplary embodiment of this application;

[0031] Figure 8 An interactive flowchart of a multimodal dialogue task processing system provided for an exemplary embodiment of this application;

[0032] Figure 9 A schematic diagram of the structure of a multimodal task processing device provided in an exemplary embodiment of this application;

[0033] Figure 10 This is a schematic diagram of the structure of a server provided in an embodiment of this application.

[0034] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0035] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0036] First, let me explain the terms used in this application:

[0037] Multimodal: Data with multiple heterogeneous modalities, including but not limited to: text, images, videos, audio, program code, etc.

[0038] MatMul or GEMM (General Matrix Multiplication): Matrix multiplication.

[0039] LayerNorm: Normalizes each single batch.

[0040] GELU: Gaussian error linear unit activation function.

[0041] Embedding: Representing a word using a vector.

[0042] Int8 quantization technology supports static quantization, training-aware quantization, and dynamic quantization. Dynamic quantization is the first recommended method as it can achieve lossless accuracy and performance acceleration without the need for algorithm engineers to fine-tune.

[0043] Subgraph fusion: Subgraphs are automatically matched to the user's computation graph to replace fusion operators, thereby merging memory accesses of many element-wise operators (such as +, -, *, / ) in the network. Element-wise operators refer to operations that require bit-by-bit execution and have a rich set of element-wise computations, such as element-wise addition, element-wise multiplication, element-wise subtraction, and finding extrema.

[0044] Operators: Operators refer to computational operations performed on data. Operators are implemented through functions based on specified parameters. Different operators are interconnected, representing data transmission relationships between them. Each operator can perform corresponding computational operations on the data input to it and output the corresponding calculation result. For example, operators represent basic arithmetic operations such as addition, subtraction, multiplication, and division; operators represent matrix transpose operations; operators represent decoding operations; and in image processing, operators represent operations such as filtering and edge detection.

[0045] Operator optimization (OPTune): Key operators usually have multiple implementations (e.g., Conv has 10+ implementations). An automatic performance evaluation is performed at runtime, and the fastest implementation is selected for inference.

[0046] Global Optimization: Using dynamic programming to compute a better combination of data formats and operators in the computational process of the network, thereby eliminating some redundant computations and memory accesses.

[0047] Multimodal tasks refer to downstream tasks that involve multiple modalities of data, such as images and text, in their input and output. Examples include visual question answering, image description, visual entailment, representation and understanding, and image generation.

[0048] Visual question answering task: Based on the input image and the question, determine the answer to the question from the visual information of the input image.

[0049] Image description task: Generate descriptive text for the input image.

[0050] Visual entailment task: Predict the semantic relevance between input images and text, i.e., entailment, neutrality, or contradiction.

[0051] The task of expression and comprehension involves locating the image region in the input image that corresponds to the input text.

[0052] Image generation task: Generate an image based on the input descriptive text.

[0053] Text-based sentiment classification task: Predict the sentiment classification information of input text.

[0054] Text summarization task: Generate a summary of the input text.

[0055] Multimodal pre-trained models refer to pre-trained models whose input and output data involve multiple modalities such as images and text. After fine-tuning and training, they can be applied to multimodal task processing.

[0056] Pre-trained language model: A pre-trained model obtained by pre-training a large language model (LLM).

[0057] Natural Language Generation (NLG) is an automated process that uses computers to generate language text under specific interactive objectives. Its main purpose is to automatically construct high-quality language text that humans can understand.

[0058] Large models refer to deep learning models with a massive number of parameters, typically containing hundreds of millions, tens of billions, or even trillions of parameters. Large models can also be called foundation models. They are pre-trained on large-scale unlabeled corpora, producing pre-trained models with hundreds of millions of parameters. These models can adapt to a wide range of downstream tasks and have good generalization ability. Examples include Large Language Models (LLMs) and multi-modal pre-training models.

[0059] In practical applications, large models only require a small number of samples to fine-tune the pre-trained model before they can be applied to different tasks. Large models can be widely used in fields such as Natural Language Processing (NLP) and Computer Vision. Specifically, they can be applied to computer vision tasks such as Visual Question Answering (VQA), Image Captioning (IC), and Image Generation, as well as NLP tasks such as text-based sentiment classification, text summarization, and machine translation. The main application scenarios for large models include digital assistants, intelligent robots, search, online education, office software, e-commerce, and intelligent design.

[0060] In recent years, natural language generation models have become an important field within artificial intelligence and natural language processing. Compared to traditional statistical language models, modern language generation models based on deep learning mostly employ end-to-end learning methods to construct relationships between words and context, thereby significantly improving the performance of generated text. In particular, pre-trained models based on Transformers can effectively capture linguistic knowledge, greatly promoting the development of natural language generation models. Natural language generation models can be implemented using large models, including multimodal pre-trained models and pre-trained language models.

[0061] Traditional Natural Language Generator (NLG) models typically use text as both input and output, and their encoding modules only support text encoding. However, with advancements in technology, simple text is no longer sufficient to demonstrate a model's capabilities. More and more models are using multimodal data such as images, audio, video, and program code for training and inference, ultimately enabling them to understand multimodal data and perform multimodal tasks such as "image description" and "summarizing video content."

[0062] However, such models still have performance issues when performing multimodal inference. Multimodal data takes up a lot of input length. For example, an image, after being encoded, often takes up the equivalent of hundreds of characters in input length, which greatly affects the overall inference performance of the model, resulting in long inference time and low performance for multimodal tasks.

[0063] This application provides a multimodal task processing method. An optimization strategy is employed to optimize the computational efficiency of operators in the model units of the initial task model to obtain a target task model. During multimodal task processing, multimodal input data to be processed is acquired and input into the target task model. The target task model encodes and fuses the multimodal input data to obtain a multimodal fused feature representation. Task processing is then performed based on this multimodal fused feature representation to generate the task processing result. This realizes an inference mode capable of accepting multimodal input. Furthermore, since the target task model used for multimodal task processing is obtained by optimizing the computational efficiency of operators in the model units of the initial task model using an optimization strategy, the computational efficiency of the model can be improved while maintaining model accuracy without degradation. This significantly shortens the multimodal task processing time and improves the efficiency and performance of multimodal task processing.

[0064] Figure 1 This is a schematic diagram of an example task processing system architecture to which this application applies. Figure 1As shown, the system architecture includes a server and endpoint devices. The server and endpoint devices have a communication link, enabling communication between them.

[0065] The server can be a server cluster deployed in the cloud or a local device with computing capabilities. The server runs a target task model for performing multimodal tasks. This target task model is obtained by optimizing the computational efficiency of the operators in the model units of the initial task model using optimization strategies. The target task model not only maintains the inference accuracy of the initial task model but also significantly improves the model's computational efficiency, shortening the inference time for multimodal tasks. The target human model can be a Natural Language Generative (NLG) model or various pre-trained large models, such as pre-trained language models or multimodal pre-trained models; no specific limitations are imposed here.

[0066] When performing a multimodal task, the server acquires the multimodal input data to be processed, inputs the multimodal input data into the target task model, and encodes and fuses the multimodal input data through the target task model to obtain a multimodal fused feature representation. The server then performs task processing based on this multimodal fused feature representation to generate the task processing result. The feature representation is obtained by encoding or fusing the input data, and is typically a tensor representation, but can also be a vector representation or other encoding result representation; no specific limitation is made here.

[0067] Edge devices are electronic devices used by users, specifically hardware devices with network communication, computing, and information display capabilities, including but not limited to smartphones, tablets, desktop computers, and servers. Users refer to individuals or organizations requesting the execution of multimodal tasks. Users provide multimodal input data to be processed to the server through edge devices and obtain task processing results from the server. Furthermore, the edge devices output the task processing results or perform subsequent processing logic based on the results to achieve various human-computer interaction functions.

[0068] In one example scenario, this solution is applied to multimodal dialogue / question-answering scenarios, implementing human-computer dialogue / question-answering based on a target dialogue model. This target dialogue model can be a Natural Language Generative (NLG) model, a pre-trained language model, a multimodal pre-trained model, etc., without specific limitations. Users input multimodal input data via their mobile devices, including but not limited to text, images, audio, video, and program code. The mobile device sends the multimodal input data to the server. The server inputs the multimodal input data into the target task model, encodes and fuses the multimodal input data to obtain a multimodal fused feature representation, and performs task processing based on this feature representation to generate a task processing result. Further, the server sends the task processing result to the mobile device. The mobile device outputs the task processing result to achieve human-computer interaction with the user. Additionally, the mobile device can also perform subsequent processing on the task processing result based on configured requirement logic and output the subsequent processing result to meet user needs and achieve various human-computer interaction functions.

[0069] Alternatively, in an optional embodiment, after generating the task processing result, the server can also execute the configured post-processing logic based on the task processing result to obtain the final output result, and send the final output result to the edge device. The edge device outputs the final output result to meet user needs and realize various human-computer interaction functions with the user.

[0070] It should be noted that the method of this application can be applied to various tasks in computer vision, natural language processing, and other fields such as machine translation, visual question answering, image description, and text summarization generation. It can also be applied to application scenarios such as digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, and open-source cloud platform management services (MaaS, Metal as a Service). When applied to different tasks, the modalities of the multimodal input data can vary, and can be configured according to the specific needs of the task scenario. This embodiment does not specifically limit the tasks and application scenarios used.

[0071] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0072] Figure 2 This is a flowchart illustrating a multimodal task processing method provided as an exemplary embodiment of this application. The execution entity in this embodiment is the server in the aforementioned system architecture. Figure 2 As shown, the specific steps of this method are as follows:

[0073] Step S21: Obtain the multimodal input data to be processed.

[0074] Multimodal input data includes, but is not limited to, data in the following modalities: text, images, audio, video, and program code. When applied to different task scenarios, the input data to be processed may include data in different modalities.

[0075] For example, in a human-computer dialogue task scenario, the multimodal input data provided by the user includes two inputs: a given image and a question text. The question text could be something like "Who is the person in the picture?". The target task model processes the multimodal input data to obtain a response text, which serves as the answer to the question text.

[0076] For example, in a text summarization task scenario, the multimodal input data from the user includes a text paragraph as input. The target task model processes the multimodal input data, and the resulting task output is a summary of the generated text paragraph.

[0077] In practical applications, there are various ways for the server to acquire the multimodal input data to be processed in this step, and the specific method selected depends on the actual situation. This embodiment does not impose any limitations on this. For example, the server can receive multimodal input data input by the user, or the user can store the multimodal input data in advance in the data storage unit of the data input server, and the server can extract the multimodal input data to be processed from the data storage unit.

[0078] Furthermore, the server can process multimodal input data for a single task request, or it can process multimodal input data for multiple task requests simultaneously; no specific limitation is made here. Multimodal input data for a single task request can include one or more inputs, and different inputs can be data of different modalities. In some scenarios, multimodal input data can also include multiple inputs of the same modality, such as multiple images or multiple text segments with different physical meanings.

[0079] Step S22: Input the multimodal input data into the target task model, encode and fuse the multimodal input data through the target task model to obtain the multimodal fusion feature representation, and perform task processing based on the multimodal fusion feature representation to generate task processing results; wherein, the target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy.

[0080] In this embodiment, the target task model can receive multimodal input data, which includes multiple inputs, and any two inputs can be data of different or the same modality.

[0081] After obtaining the multimodal input data from the user, the multimodal input data is input into the target task model. The target task model encodes and fuses the multimodal input data to obtain the multimodal fused feature representation. The task processing is then performed based on the multimodal fused feature representation to generate the task processing result, thereby realizing multimodal task processing.

[0082] For example, in the aforementioned human-computer dialogue task scenario, the multimodal input data from the user includes a given image and a question text. The given image and question text are input into the target task model. The target task model encodes and fuses the given image and question text separately, obtaining a multimodal fused feature representation of the image and text. Further, inference is performed based on the multimodal fused feature representation to achieve multimodal task processing, generating a response text to the question text, thus obtaining the task processing result.

[0083] It should be noted that if the multimodal input data only contains text input, that is, if there is no non-text modal input in the multimodal input data, the solution of this embodiment can also be used. The text input is directly input into the target task model, the target task model encodes the text input to obtain the feature representation of the text input, and the task processing is performed according to the feature representation of the text input to generate the task processing result.

[0084] The solution in this embodiment is not limited to the type and representation of multimodal input data. Whether it is existing images, videos, or voice, or more data types / modalities that may appear in the future, this inference process can be used to accelerate model inference, which has strong versatility.

[0085] In this embodiment, the target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy. Specifically, the target task model can be obtained by updating and setting the operators of each model unit based on the operator information of each model unit in the initial task model using a corresponding computational efficiency improvement strategy. Since the target task model is obtained by updating and setting the operators of each model unit based on the operator information of each model unit in the initial task model using a corresponding computational efficiency improvement strategy, the computational efficiency of the model can be improved while maintaining the original accuracy of the initial task model without degradation, greatly shortening the task processing time and improving the efficiency and performance of task processing. Different types of operators may use different computational efficiency improvement strategies.

[0086] For example, for operators of arithmetic operation type in each model unit, multiple operators of arithmetic operation type in the same model unit can be fused to construct the target operator.

[0087] For example, for operators of the transpose operation type in each model unit, a target operator can be constructed according to the pre-set stride parameters. The target operator is used to store target feature data into memory according to the stride parameters and read target feature data from memory when calculating the target feature data. The target feature data is the feature data for the transpose operation.

[0088] For example, for activation type operators in each model unit, the activation type operators can be postponed to the execution process of the target operator, and the execution logic of the target operator can be updated. This can reduce the overhead of launching the kernel, while achieving a balance between accuracy and speed.

[0089] For example, for the decoding type operator in each model unit, the decoding result corresponding to the decoding type operator can be obtained; the decoding result can be stored in a pre-set decoding cache unit; the decoding logic of the operator can be updated according to the decoding result in the decoding cache unit to obtain the target operator, which can avoid unnecessary repeated calculations during the decoding process and improve the model operation speed.

[0090] In an optional embodiment, in step S22 above, when encoding and fusing the multimodal input data through the target task model to obtain the multimodal fused feature representation, each input in the multimodal input data can be encoded separately to obtain the feature representation corresponding to each input. Further, the feature representations corresponding to each input are concatenated to achieve the fusion of feature representations of different modal inputs, thus obtaining the multimodal fused feature representation.

[0091] In another alternative embodiment, such as Figure 3 As shown, in step S22 above, the multimodal input data is encoded and fused through the target task model to obtain the multimodal fused feature representation, which can be implemented using the following steps S31-S34:

[0092] Step S31: Encode the non-textual modal inputs contained in the multimodal input data to obtain the feature representations of the non-textual modal inputs.

[0093] In this step, the non-textual modal inputs in the multimodal input data are encoded separately to obtain the feature representations of each non-textual input.

[0094] Encoding inputs of different modalities can be achieved using different encoding algorithms / models. For example, for images in multimodal input data, an image encoding model is used to encode each input image separately, obtaining a feature representation of the input image. For audio in multimodal input data, an audio encoding model / algorithm is used to encode each input audio separately, obtaining a feature representation of the input audio. For video in multimodal input data, a video encoding model / algorithm is used to encode each input video separately, obtaining a feature representation of the input video. Specifically, the encoding model / algorithm for each input modality can be selected based on experience. For any modality, the commonly used encoding model / algorithm implementation for that modality of data can be selected based on experience; no specific limitations are made here.

[0095] In practical applications, a server can process multimodal input data from a single task request, or it can process multimodal input data from multiple task requests simultaneously; no specific limitation is made here. Multimodal input data for a single task request can include one or more inputs, and different inputs can be data from different modalities. In some scenarios, multimodal input data can also include multiple inputs of the same modality; for example, a task request can input multiple images or multiple pieces of text with different physical meanings.

[0096] In one optional embodiment, a modality identifier can be set for each modality to distinguish and refer to different modalities. For example, different modalities can be numbered, and the corresponding number for each modality can be used as the modality identifier. For instance, the image identifier can be set to 10001, the video identifier to 10002, the audio identifier to 10003, and so on. Furthermore, the modality identifier for each modality can also be represented by different characters / strings. The modality identifier for each modality can be set according to specific application scenarios and experience, and is not specifically limited here.

[0097] After obtaining the feature representation of the non-text modal input, the sequence of feature representations of at least one input of the same non-text modality in the multimodal input data is stored in correspondence with the modality identification information of the non-text modality.

[0098] For example, suppose the multimodal input data includes text A, image B, image C, audio D, and audio E. The image number can be set to 10001, and the audio number can be set to 10003. Then, after encoding image B and image C respectively to obtain the feature representation of image B and the feature representation of image C, and encoding audio D and audio E respectively to obtain the feature representations corresponding to audio D and audio E, for the image modality, according to the order in which they appear in the multimodal input data, the feature representations of the two images B and C are arranged into a feature representation sequence for the image, and this feature representation sequence is stored in correspondence with the modal identifier information 10001 corresponding to the image. For the audio modality, according to the order in which they appear in the multimodal input data, the feature representations of the two audios D and E are arranged into a feature representation sequence for the audio, and this feature representation sequence is stored in correspondence with the modal identifier information 10003 corresponding to the audio, resulting in the storage content shown in Table 1 below.

[0099] Table 1

[0100] Modal identification information Sequences represented by features 10001 {Feature representation of image B; Feature representation of image C} 10003 {Feature representation of audio D; Feature representation of audio E}

[0101] Optionally, the sequence of feature representations for any modality can be stored in a list.

[0102] By storing the sequence of feature representations of the input for each modality in correspondence with modality identification information, and by traversing the feature representations in the sequence of feature representations corresponding to each modality, the feature representations in the subsequent step S34 can be replaced, thereby improving the processing efficiency of obtaining feature representations for multimodal fusion.

[0103] Step S32: Replace the non-text modal inputs in the multimodal input data with the corresponding modal identifier information to obtain plain text input.

[0104] In this step, the non-text modal inputs in the original multimodal input data are replaced with the modal identifier information of the corresponding modality. The replaced input data contains only data of a single text modality, resulting in plain text input.

[0105] Furthermore, if the multimodal input data for the same task request includes multiple inputs of a certain modality, then each input of that modality will be replaced with its modality identifier information. For example, if a multimodal input data set contains two or more input images, then the modality identifier information of each image will be replaced sequentially.

[0106] For example, the non-text modal input in the multimodal input data is replaced with a text fragment of the same length, which is concatenated using the corresponding modality identifier information, to obtain the plain text input. This keeps the length of the original input unchanged, which facilitates the replacement of feature representations in the subsequent step S34 and improves the processing efficiency of obtaining the feature representation of multimodal fusion.

[0107] For example, modality numbers can be used as modality identification information, with one modality number occupying one character. Assume an image's number can be 10001. For images in the multimodal input data, if an image occupies 384 characters, then the images in the multimodal input data will be replaced with 384 instances of 10001.

[0108] In addition, for images in multimodal input data, when encoding image data, a preset encoding interface can be used to convert the Vision Transformer (VIT) model used for image encoding into an Open Neural Network Exchange (ONNX) model through a model conversion service, which can improve the efficiency and performance of image encoding.

[0109] The input encoding process is divided into character encoding and positional encoding. Character encoding involves calculating the feature representation (tensor representation) corresponding to a single symbol (such as a character in text or a pixel block in an image) from the input data. This process itself is relatively stable. Positional encoding calculates the tensor offset based on the position of the character in the input and adds it to the feature representation (tensor representation) after character encoding.

[0110] Step S33: Encode the plain text input to obtain the feature representation of the plain text input.

[0111] In this step, the plain text input is encoded using a text encoding model / algorithm to obtain a feature representation of the plain text input. The text encoding model / algorithm used can be selected based on the needs of the actual application scenario, and no specific limitations are made here.

[0112] In this step, the encoding of plain text input can also be achieved by calling the preset high-efficiency encoding interface, thereby maintaining the performance and efficiency of model inference.

[0113] Step S34: Use the feature representation of the non-text modal input to replace the feature representation of the corresponding modality identification information in the feature representation of the plain text input, and obtain the feature representation of multimodal fusion.

[0114] To achieve the fusion of feature representations of multimodal inputs, this step involves replacing the feature representations of corresponding modality identification information in the feature representations of plain text inputs with the feature representations of non-text modal inputs, based on the feature representations of plain text inputs, to obtain the multimodal fused feature representations.

[0115] Specifically, for any feature representation of a non-text modal input, based on the length of the feature representation of the non-text modal input, a text fragment of the same length, formed by concatenating the corresponding modal identifier information, is found in the plain text input to determine the position of the text fragment in the plain text input; based on the position of the text fragment in the plain text input, the content at the corresponding position in the feature representation of the plain text input is replaced with the feature representation of the non-text modal input to obtain the feature representation of multimodal fusion.

[0116] In addition, before performing feature representation replacement for non-text modal input in this step, it can be checked whether there is a feature representation for non-text modal input. If there is no feature representation for non-text modal input, the feature representation for plain text input is directly used as the final feature representation for subsequent task processing to obtain the task processing result.

[0117] If feature representations for non-text modal inputs exist, all feature representations for non-text modal inputs are traversed, and the modality identifier information corresponding to the stored feature representation of that modal input is parsed out. If the multimodal input data includes multiple inputs of a certain modality, that is, the modality recognition information of that modality corresponds to multiple feature representations, the feature representations of that modal input are parsed out sequentially. In the plain text input, the first text fragment formed by concatenating the corresponding modality identifier information is found. It is determined whether the length of the text fragment formed by concatenating the modality identifier information is the same as the length of the feature representation of that modal input. If they are the same, the text fragment is replaced with the feature representation of that modal input, and then the replacement of the next feature representation is performed, until all feature representations of non-text modal inputs have been replaced, resulting in the multimodal fusion feature representation.

[0118] If no text fragment composed of the corresponding modality identifier information is found in the plain text input, or if the length of the text fragment composed of the found modality identifier information is different from the length of the feature representation of the modality input, an error message can be output to indicate that the task execution has failed.

[0119] In one optional embodiment, a multimodal fusion operator can be used to replace the feature representation of the corresponding modality identification information in the feature representation of the plain text input with the feature representation of the non-text modal input, thus obtaining a multimodal fused feature representation. Specifically, all feature representations of the non-text modal input, the replaced plain text input, and the feature representation of the plain text input are input to the multimodal fusion operator. The multimodal fusion operator uses the feature representation of the non-text modal input to replace the feature representation of the corresponding modality identification information in the feature representation of the plain text input, thus obtaining a multimodal fused feature representation. The multimodal fusion operator then outputs the multimodal fused feature representation. Alternatively, if no feature representation of the non-text modal input is input, the multimodal fusion operator directly outputs the feature representation of the plain text input. In the case where no feature representation of the non-text modal input exists, i.e., no non-text modal input is provided, the feature representation of the plain text input is the same as the feature representation of the original input text.

[0120] In real-world scenarios like online conversations, the emergence of multimodal responses is often unpredictable. For example, a user might suddenly insert an image during a chat. Therefore, multimodal understanding must be routinely supported by the inference engine. The advantage of this solution is that it can routinely support multimodal input while still ensuring that the inference performance in pure text input scenarios does not degrade. In other words, introducing multimodality will not affect the original pure text inference performance.

[0121] The method in this embodiment encodes the non-text modal inputs contained in the multimodal input data to obtain feature representations of the non-text modal inputs; replaces the non-text modal inputs in the multimodal input data with the corresponding modality identification information to obtain plain text inputs; encodes the plain text inputs to obtain feature representations of the plain text inputs; and replaces the feature representations of the corresponding modality identification information in the feature representations of the plain text inputs with the feature representations of the non-text modal inputs to obtain the feature representations of multimodal fusion, thereby realizing the fusion of multimodal inputs. This method can support scenarios where the input data contains only text content and does not contain non-text modal content, achieving efficient processing of multimodal tasks while preserving the inference performance of the model in plain text.

[0122] In practical applications, the input encoding process is divided into character encoding and positional encoding. Character encoding involves calculating the feature representation (tensor representation) corresponding to a single symbol (such as a character in text or a pixel block in an image) from the input data. This process itself has relatively stable performance. Positional encoding calculates the tensor offset based on the position of the character in the input and adds it to the feature representation (tensor representation) after character encoding.

[0123] As the number of user task requests increases, the inference engine cannot fully utilize the hardware performance by processing only one request at a time. Therefore, multiple requests will be grouped together for inference. In the process of grouping requests, it is inevitable that different requests will have different input lengths.

[0124] Currently, in model inference schemes (such as PyTorch models), this situation requires padding placeholders at the end of the shorter input to make the input length of all requests the longest one, thus aligning the input lengths of all requests before proceeding to the next encoding and inference step. In this scheme, when the input lengths within a group of requests differ significantly, it leads to a large amount of invalid placeholder padding and invalid computation. For example, grouping a request with an input length of 20 and a request with an input length of 1000 together and padding them with placeholders actually encodes two requests with an input length of 1000, wasting a significant amount of computational resources. There are also problems with positional encoding calculations. For instance, in a single inference run, the next generated word should be at position 21 for a request with an input length of 20. However, based on the existing model inference scheme, due to placeholder padding and the need for uniform length, the positional encoding of subsequent generated characters is calculated directly from 1001, resulting in inconsistent encoding results between single runs and combined runs, affecting the stability of text generation.

[0125] In an optional embodiment, the multimodal input data to be processed includes multimodal input data of multiple task requests, which constitute a set of requests. During the encoding of plain text input in step S33, the plain text input of each task request is passed to a preset encoding interface. The maximum length of the plain text input of each task request is set through the preset encoding interface. By padding placeholders at the end of plain text inputs shorter than the maximum length, the lengths of the plain text inputs of multiple task requests are aligned. The aligned plain text inputs of multiple task requests are then encoded. The placeholders can be selected based on actual application scenarios and experience, and are not specifically limited here.

[0126] Furthermore, when encoding aligned plain text input from multiple task requests, the placeholders contained in the aligned plain text input are parsed to determine the actual effective length of the aligned plain text input and the placeholder offsets of the contained placeholders; based on the actual effective length of the aligned plain text input and the placeholder offsets of the contained placeholders, the actual effective input from the aligned plain text input of multiple task requests is encoded.

[0127] Specifically, the plain text input of a group of requests is parsed separately, and the placeholder padding of the plain text input of each request is scanned to calculate the actual effective length of the plain text input of each request (the number of non-placeholder symbols), and to obtain the placeholder offset of the plain text input of each request. The placeholder offset of the aligned plain text input is the actual length of the plain text input minus the uniform length of the plain text input of the group of requests after padding; the placeholder offset of the plain text input can be negative or zero.

[0128] Furthermore, during the encoding process, the feature representation corresponding to each character in the aligned plain text input is calculated sequentially. During the execution of the kernel, it can be determined which input the currently calculated symbol belongs to. If the position of the symbol is greater than the actual effective length of the input, no calculation is performed, and the feature representation of the character is padded with 0.

[0129] Furthermore, new characters will be continuously generated during the model's subsequent inference process, and these new characters also need to be encoded. The encoding process involves positional encoding calculations. For each newly generated character, its actual position is the current position plus a placeholder offset.

[0130] For example, consider a set of requests containing plain text input of length 20 and plain text input of length 1000. The aligned plain text input has the same length of 1000. For the plain text input of length 20, the placeholder offset after alignment is -980. When calculating the position encoding of the first newly generated character, the current position is 1001. Adding the placeholder offset -980, we get the final actual position 21. This result is the same as the result of encoding only for this request, ensuring the stability of the text generation effect.

[0131] In this embodiment, when encoding multimodal input data, although placeholders are used to fill request inputs of different lengths to a uniform length through a preset encoding interface, the encoding is performed according to the actual effective length of each request input during the actual encoding process. This approach can avoid invalid calculations of filling placeholders during the encoding process while processing multiple task request inputs in parallel, thereby improving the encoding efficiency and performance in multi-task request scenarios and thus improving the overall efficiency and performance of task inference.

[0132] In one optional embodiment, the construction of the target task model can be divided into the following three steps:

[0133] Model format conversion and loading: The initial task model is obtained and parsed by a model converter, transforming it into an internal representation format that the server can process, such as .as or .asparam files. The .as file stores the model representation graph structure, while the .asparam file stores specific parameters, i.e., operator weights. Further, a model parser can be used to read and parse the initial task model file in its internal representation format to obtain the model protocol. Based on the configuration protocol within the model protocol, a model instance is generated. Using the operators in the model instance as nodes, a model representation graph corresponding to the initial task model is constructed, and the operator information for each operator in the model representation graph is identified.

[0134] Model optimization: The graph optimizer adjusts the model instance and generates an optimized internal model representation (AsModel). In other words, it uses the computational efficiency improvement strategies corresponding to the operator information to update and set each operator to obtain the target operator.

[0135] Loading model parameters: The graph executor calls the optimized target operator to execute each operator node in the model's internal representation sequentially, constructing the target task model. Furthermore, a distributed scheduler can be used to distribute the execution of each operator node in the model's internal representation, improving the efficiency of model optimization.

[0136] It should be noted that after the graph executor obtains the target task model, it can process the input task (Input tensor) and output the corresponding task processing result (Output tensor). Users can also configure the .yml config file to adjust the model's functionality according to their needs.

[0137] Specifically, such as Figure 4 As shown, the target task model can be obtained by optimizing the computational efficiency of the operators in the model units of the initial task model through the following steps S41-S45 using an optimization strategy:

[0138] Step S41: Analyze the initial task model to obtain the operators of multiple model units in the initial task model.

[0139] Specifically, the initial task model refers to a model with task processing capabilities, such as a recurrent neural network (RNN), a convolutional neural network (CNN), a pre-trained large model, a multimodal pre-trained model, etc. The specific model to be selected depends on the actual situation, and this embodiment does not impose any limitations on it. The multiple model units in the initial task model can also be understood as the various processing layers in the initial task model, such as normalization layers, attention layers, etc.

[0140] Step S42: Using each operator as a node, construct the model representation diagram corresponding to the initial task model.

[0141] Step S43: Identify the operator information of each operator in the model representation diagram.

[0142] Step S44: Utilize the computational efficiency improvement strategy corresponding to the operator information to optimize the computational efficiency of each operator and obtain the target operator.

[0143] In the first optional implementation of step S44, the first operators of the target model unit can be fused to construct the target operator. Here, the first operator of the target model unit is an arithmetic operation, and the target model unit is any model unit in the initial task model. This fully utilizes the continuity of CPU and GPU memory read / write operations, significantly reduces the overhead of operators in input / output, and saves some redundant computations.

[0144] For example, the element-wise multiplication (MatMul) and element-wise addition (BiasAdd) operators can be merged into a single GEMM operation; multiple transpose (Transpose), multiple multiplication (BatchMatMul), and multiplication (Mul) operators can be merged into a single BatchGEMM operation; and addition (Add) and normalization (Softmax) operators can be merged into a single masked normalization (MaskedSoftmax) operation. It should be noted that the server has undergone comprehensive updates and optimizations to the GEMM operators. Specific methods include, but are not limited to, using FP16 and BF16 data formats to increase throughput, using INT8 for dynamic quantization, using the NC4HW4 data layout format, fully utilizing the concurrent read / write capabilities of vector registers, using a parallel computing framework, hand-writing an assembly GEMM kernel for the CPU version, and fully utilizing L1 cache (first-level cache), L2 cache (second-level cache), vector registers, and concurrent pipelines. The specific methods are selected based on the actual situation, and this embodiment does not impose any limitations on them.

[0145] In the second optional implementation of step S44, the operator information includes the transpose operation type. The server can obtain pre-set stride parameters; construct a target operator based on the stride parameters, wherein the target operator is used to store target feature data into memory according to the stride parameters, and read target feature data from memory when calculating the target feature data, the target feature data being the feature data for the transpose operation.

[0146] Specifically, the stride parameter is a pre-set one-dimensional vector of length 4. It takes the form [a, x, y, z], representing [batch stride, horizontal stride, vertical stride, and channel stride], respectively. The general form of the stride parameter is [1, x, y, 1]. The first 1 indicates that the stride in the batch dimension is 1, meaning no samples are skipped. x represents the horizontal stride of the convolution kernel. y represents the vertical stride of the convolution kernel. The last 1 indicates that the stride in the channel dimension is 1, meaning no color channels are skipped. The stride parameter is pre-set based on a matrix calculated using GEMM. `batch` represents the number of data sets.

[0147] For example, in the calculation of attention, a two-dimensional matrix needs to be reshaped into a three-dimensional matrix, then transposed into a locally continuous memory layout before matrix multiplication. In this optional implementation, a stride parameter can be added to implement the stride_GEMM operator. Since traditional GEMMs can only access memory sequentially, reshape and transpose are required to reshape and reorganize the original matrix before calculation. stride_GEMM, however, can jump to a specified memory location for calculation by providing the corresponding stride parameter, enabling memory-skipping GEMMs. This allows direct calculation using the original matrix, saving the overhead of the reshape and transpose operators and improving model computational efficiency. The reshape operator is used to rearrange the data arrangement of elements.

[0148] In the third optional implementation of step S44, a pre-set target matrix can be obtained when the operator is of a specified form, wherein each column of the target matrix has the same number of non-zero elements; based on the target matrix, a target operator is constructed. Since common scientific computing is not designed for sparse GEMMs, the computation pattern is generally dense(A)*sparse(B) = dense(C), requiring many transpose operations in the implementation. Therefore, in this optional implementation, the server includes a sparse format and its corresponding sparse kernel, supporting the computation pattern sparse(A)*dense(B) = dense(C), avoiding the additional overhead of transpose operations. Here, A, B, and C are matrices in matrix multiplication, representing matrix A multiplied by matrix B to obtain matrix C.

[0149] For example, the server aligns memory addresses using padding (data padding characters) for traditional column-major (CSC) and row-major (CSR) matrices, employing efficient memory access instructions. Each warp (platform resource scheduling unit) computes one element, ensuring contiguous memory access. A global cache is used to cache each row of matrix A, and an L1 cache (Level 1 cache) is used to cache each column of matrix B (CSR / CSC format), fully utilizing the CPU's caching capabilities. Furthermore, the server defines a custom efficient sparse format, ELL, for certain model structures. This format is suitable for operators of a specified form, where each column of the matrix has the same number of non-zero elements, and the distribution of non-zero elements within each column is completely random. Since the target matrix B has the same number of non-zero elements in each column, load imbalance is avoided. Using a global cache to cache each row of matrix A and registers to cache each column of matrix B fully utilizes caching capabilities under non-limited input scales while improving parallelism. Simultaneously, each thread calculates one element, performing thread accumulation. Each warp calculates 32 adjacent elements, ensuring continuous memory access while avoiding the overhead of warp reduction. Furthermore, the server framework automatically selects the appropriate matrix based on performance. This avoids load imbalance issues and saves on the additional overhead of transpose operations.

[0150] In the fourth optional implementation of this step, the operator information includes the normalization type, and the server obtains the pre-set out-of-order parameters; a target operator is constructed based on the out-of-order parameters, wherein the target operator is used to perform parallel reduction of the normalized feature data according to the out-of-order parameters, and to perform operations on the normalized feature data after parallel reduction, which can reduce overhead and improve the model running efficiency.

[0151] For example, before data reduction, operators used to operate on normalized feature data include averaging (ReduceMean), subtraction (Sub), exponentiation (Pow), addition (Add), square root (Sqrt), division (Div), and multiplication (Mul). In this embodiment, after data reduction, for the layer normalization (LayerNorm) operator, the server uses a single kernel to directly implement it: small kernel fusion reduces the overhead of launching the kernel, optimizes the reduce method, uses shared memory, and uses warp-level shuffle primitives to accelerate reduce computation, that is, constructs the target operator based on the out-of-order parameter (shfl_down_sync). Here, reduce is a form of continuous identical operations such as accumulation or multiplication. Data in the same warp can be exchanged at high speed through shuffle instructions.

[0152] In the fifth optional implementation of this step, the operator information includes the activation type. After the server appends the activation type operator to the execution process of the target operator, it updates the execution logic of the target operator, which can reduce the overhead of launching the kernel and achieve a balance between accuracy and speed.

[0153] For example, several approaches can be used to update the GELU operator. One approach is to directly use kernel-fused serially executed activation type operators and GEMM, that is, to directly postpone activation type operators such as GELU and ReLU into the GEMM computation process, allowing the activation type operators and GEMM to be computed simultaneously. It should be noted that the specific activation type operator can be selected based on the downstream tasks of the model.

[0154] In the sixth optional implementation of this step, the operator information includes the decoding type. The server obtains the decoding result corresponding to the operator of the decoding type; stores the decoding result in a pre-set decoding cache unit; and updates the decoding logic of the operator based on the decoding result in the decoding cache unit to obtain the target operator. This can avoid unnecessary repetitive calculations during the decoding process and improve the model's computation speed.

[0155] For example, before operator optimization, the decoding process involves many repetitive calculations. Each decoding step requires recalculating the vector representation corresponding to the previously generated token (character): when generating id1 in the first step, generating id2 based on id1 in the second step, and generating id3 based on id2 in the third step, both id1 and id2 need to be generated. Therefore, a decoding cache unit can be added inside the server to record the decoding result of each step, eliminating unnecessary repetitive calculations.

[0156] In the seventh optional implementation of this step, the server can store the weights corresponding to the target operator according to a specified weight storage format. This specified weight storage format includes operator attribute information and model runtime data. Specifically, the specified weight storage format is a binary weight storage format. The operator attribute information includes basic information such as weight name, data type, and shape, as well as special information such as sparsity type and splitting type. The model runtime data refers to the specific numerical values ​​of the weights.

[0157] For example, the weights and model representation graph corresponding to the target operator can be stored in the same .as file. Another possible implementation in this specification is to store the weights and model representation graph separately; that is, the .as file stores the model representation graph structure, and the .asparam file stores the weights corresponding to the target operator. Separate storage is generally used when loading very large-scale models to prevent memory overflow caused by loading a large amount of temporarily useless weight data during the model structure parsing stage.

[0158] Step S45: Construct the target task model based on the target operator.

[0159] In this step, the weights of the target operator are divided according to multiple operator execution units to obtain the division weights corresponding to each operator execution unit; the execution results obtained by each operator execution unit executing the target operator according to the division weights are received; and the execution results of each operator execution unit are merged to obtain the target task model.

[0160] For example, in one implementation of operator weight splitting, model performance can be further optimized by fusion using the Attention operator on top of model parallelism. Originally, the weights on each card were split separately from weightQ, weightK, and weightV. In this embodiment, these three weights can be fused into weightQKV, allowing the splitting of the three weights to be performed simultaneously, reducing the overhead of the operator in input and output, and avoiding some redundant computations. Specifically, the weights before fusion are the memory blocks corresponding to Q, K, and V in the upper half of the diagram. For example, Q corresponds to Q1+Q2, which are not separated in a single-card scenario. In a multi-card scenario, a weightQKV needs to be assigned to each card. (Q1, K1, V1) can be extracted as the weightsQKV of the first graphics card, and so on, extracting (Q2, K2, V2) as the weightsQKV of the second graphics card. Here, weightQ, weightK, and weightV are the names of the three weights in the model.

[0161] It's important to note that the model splitting process is handled internally by the Distributed Scheduler. Simply provide the original weight file and specify the number of GPUs to run on, and the splitting will be completed automatically, eliminating the need for manual splitting by the user and preventing the split weights from only being able to run on a fixed number of GPUs. The split data is saved as a binary file in an internal format that can be processed by a microcomputer. Subsequent runs of the same model can directly read this data without re-splitting, reducing model startup time.

[0162] For example, in one example of operator weight splitting, the input undergoes two GEMM calculations, assuming X graphics cards perform the computation in parallel. By vertically splitting weight1 from the first GEMM into X parts, each graphics card obtains a vertical result inner_i, which, when merged, yields the actual result inner. Then, the weight2 from the second GEMM is horizontally split into X parts, resulting in a horizontal result weight2_i. Performing GEMM(inner_i, weight2_i) on each graphics card then yields an output component out_i of the same size as the actual output matrix. Finally, the all reduce_sum operation is used to add up all out_i values, resulting in the final output. Here, weight1 and weight2 represent the weights from the two GEMM calculations, respectively.

[0163] For example, in another example of operator weight splitting, the input undergoes two GEMM calculations, assuming X GPUs are computing in parallel. By vertically splitting weight1 in the first GEMM into X parts, each GPU will obtain a vertical result inner_i, which, after merging, also yields the actual result inner. Then, the weight2 in the second GEMM is vertically split into X parts, resulting in a horizontal result weight2_i. Executing GEMM(inner_i, weight2_i) on each GPU will then produce an out_i of the same size as the actual output matrix. Finally, the all-gather operation is used to merge all out_i to obtain the final output.

[0164] The solution implemented in the embodiments of this specification supports the simultaneous use of multiple graphics cards on a single device for model inference, solving the problem of insufficient video memory on a single graphics card and enabling performance acceleration by utilizing weight partitioning strategies and the resources of multiple graphics cards.

[0165] The solution in this embodiment involves parsing the initial task model to obtain operators for multiple model units within the initial task model; constructing a model representation graph corresponding to the initial task model using each operator as a node; identifying the operator information for each operator in the model representation graph; optimizing the computational efficiency of each operator using computational efficiency improvement strategies corresponding to the operator information to obtain the target operator; and constructing the target task model based on the target operator. Since the target task model is obtained by optimizing the computational efficiency of the operators of each model unit based on the operator information of each model unit in the initial task model using corresponding computational efficiency improvement strategies, the computational efficiency of the task model can be improved while maintaining the processing accuracy of the task model, significantly shortening the task processing time and improving the overall efficiency of task processing.

[0166] For example, Figure 5 The architecture diagram of the multimodal task processing method provided in this application is as follows: Figure 5 As shown, taking the multimodal input data, which includes images and the text "Who is the person in the picture?", as an example, the images are encoded into feature representations, and the images in the multimodal input data are replaced with the corresponding modality identifier information 10001 to obtain plain text input. The plain text input is then encoded to obtain its feature representation. Furthermore, multimodal fusion is achieved through image feature representation replacement to obtain a multimodal fused feature representation. Finally, efficient inference is performed based on the multimodal fused feature representation to obtain the task processing result.

[0167] This application provides a multimodal task processing system, including a server and an end-side device. Figure 6 This is an interactive flowchart of a multimodal task processing system provided as an exemplary embodiment of this application. Figure 6 As shown, the interaction process between the server and the end-device in a multimodal task processing system is as follows:

[0168] Step S61: The end-side device receives the multimodal input data to be processed from the user.

[0169] The multimodal input data entered by the user in this step is similar to the multimodal input data obtained by the server in step S21 above. For details, please refer to the relevant content in the previous embodiment, which will not be repeated here.

[0170] Step S62: The end device sends multimodal input data to the server.

[0171] Step S63: The server receives multimodal input data sent by the terminal device.

[0172] Step S64: The server inputs the multimodal input data into the target task model, encodes and fuses the multimodal input data through the target task model to obtain the multimodal fused feature representation, and performs task processing based on the multimodal fused feature representation to generate task processing results; wherein, the target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy.

[0173] The specific implementation method of this step is the same as that of the aforementioned step S22. For details, please refer to the relevant content of the aforementioned embodiments, which will not be repeated here.

[0174] Step S65: The server sends the task processing result to the end device.

[0175] Step S66: The end device receives the task processing result and outputs the task processing result.

[0176] This embodiment provides an interactive flowchart of a multimodal task processing system, which can achieve efficient multimodal task processing and improve the inference efficiency and performance of multimodal tasks.

[0177] Figure 7 This is a flowchart illustrating a multimodal dialogue task processing method provided as an exemplary embodiment of this application, applied to a server in a human-computer dialogue scenario. For example... Figure 7 As shown, the specific steps of this method are as follows:

[0178] Step S71: Receive the dialogue task request sent by the receiving end device. The dialogue request includes multimodal input data.

[0179] This embodiment applies to a human-computer dialogue scenario. The user interacts with the server through a terminal device, sending a dialogue task request to the server. This dialogue task request includes multimodal input data. The server obtains the multimodal input data from the received dialogue task request. The server generates a response text based on the multimodal input data and sends it to the terminal device.

[0180] The multimodal input data included in the dialogue task request is multimodal data input and / or specified by the user through the terminal device. It is similar to the multimodal input data obtained by the server in step S21 above. For details, please refer to the relevant content in the previous embodiment, which will not be repeated here.

[0181] Step S72: Input the multimodal input data into the target dialogue model, encode and fuse the multimodal input data through the target dialogue model to obtain the multimodal fused feature representation, and perform dialogue task processing based on the multimodal fused feature representation to generate response text; wherein, the target dialogue model is obtained by optimizing the computational efficiency of the operators of the model units in the initial dialogue model using an optimization strategy.

[0182] The initial dialogue model can be a large model that implements human-computer dialogue, such as a pre-trained language model or a multimodal pre-trained model; no specific limitations are made here.

[0183] The specific implementation method of this step is the same as that of the aforementioned step S22. For details, please refer to the relevant content of the aforementioned embodiments, which will not be repeated here.

[0184] Step S73: Send a response text to the end-side device.

[0185] In this embodiment, the target dialogue model is obtained by optimizing the computational efficiency of the operators of the model units in the initial dialogue model using an optimization strategy. Specifically, the target dialogue model can be obtained by updating and setting the operators of each model unit based on the operator information of each model unit in the initial dialogue model using a corresponding computational efficiency improvement strategy. Since the target dialogue model is obtained by updating and setting the operators of each model unit based on the operator information of each model unit in the initial dialogue model using a corresponding computational efficiency improvement strategy, the computational efficiency of the dialogue model can be improved while maintaining the inference accuracy of the dialogue model without degradation. This significantly shortens the processing time of the multimodal dialogue task and improves the efficiency and performance of the multimodal dialogue task.

[0186] In an optional embodiment, in step S72, the multimodal input data is encoded and fused using the target dialogue big model to obtain a multimodal fused feature representation. Specifically, this can be achieved in the following way:

[0187] The non-text modal inputs included in the multimodal input data are encoded separately to obtain the feature representation of the non-text modal inputs; the non-text modal inputs in the multimodal input data are replaced with the corresponding modality identification information to obtain the plain text inputs; the plain text inputs are encoded to obtain the feature representation of the plain text inputs; the feature representation of the non-text modal inputs is used to replace the feature representation of the corresponding modality identification information in the feature representation of the plain text inputs to obtain the feature representation of the multimodal fusion.

[0188] This implementation method is consistent with the implementation method of the aforementioned steps S31-S34. For details, please refer to the relevant content of the aforementioned embodiments. This embodiment will not repeat the details here.

[0189] This application provides a multimodal dialogue task processing system, including a server and an end-side device. Figure 8 An interaction flowchart of a multimodal dialogue task processing system provided as an exemplary embodiment of this application. Figure 8 As shown, the interaction flow between the server and the client device in a multimodal dialogue task processing system is as follows:

[0190] Step S81: The terminal device receives a dialogue task request sent by the terminal device. The dialogue task request includes multimodal input data.

[0191] The multimodal input data entered by the user in this step is similar to the multimodal input data obtained by the server in step S71 above. For details, please refer to the relevant content in the previous embodiment, which will not be repeated here.

[0192] Step S82: The end device sends multimodal input data to the server.

[0193] Step S83: The server receives multimodal input data.

[0194] Step S84: The server inputs the multimodal input data into the target dialogue model, encodes and fuses the multimodal input data through the target dialogue model to obtain the multimodal fused feature representation, and performs dialogue task processing based on the multimodal fused feature representation to generate response text; wherein, the target dialogue model is obtained by optimizing the computational efficiency of the operators of the model units in the initial dialogue model using an optimization strategy.

[0195] The specific implementation method of this step is the same as that of the aforementioned step S22. For details, please refer to the relevant content of the aforementioned embodiments, which will not be repeated here.

[0196] Step S85: The server is also used to send a reply text to the end device.

[0197] Step S86: The end-side device is also used to receive and output a response text.

[0198] This embodiment provides an interactive flowchart of a multimodal dialogue task processing system, which can efficiently process multimodal dialogue tasks and improve the inference efficiency and performance of multimodal dialogue tasks.

[0199] Figure 9 This is a schematic diagram of the structure of a multimodal task processing apparatus provided in an exemplary embodiment of this application. The multimodal task processing apparatus provided in this embodiment can execute the server processing flow in the multimodal task processing method embodiment. Figure 9 As shown, the multimodal task processing device 90 includes: a multimodal data acquisition module 91, a multimodal task processing module 92, and a model optimization module 93.

[0200] The multimodal data acquisition module 91 is used to acquire the multimodal input data to be processed.

[0201] The multimodal task processing module 92 is used to input multimodal input data into the target task model, encode and fuse the multimodal input data through the target task model to obtain a multimodal fused feature representation, and perform task processing based on the multimodal fused feature representation to generate task processing results. The target task model is obtained by optimizing the computational efficiency of the operators in the model units of the initial task model using an optimization strategy.

[0202] The model optimization module 93 is used to optimize the computational efficiency of the operators of the model units in the initial task model using optimization strategies to obtain the target task model.

[0203] In an optional embodiment, when encoding and fusing multimodal input data to obtain a multimodal fused feature representation, the multimodal task processing module 92 is further configured to:

[0204] The non-text modal inputs included in the multimodal input data are encoded separately to obtain the feature representation of the non-text modal inputs; the non-text modal inputs in the multimodal input data are replaced with the corresponding modality identification information to obtain the plain text inputs; the plain text inputs are encoded to obtain the feature representation of the plain text inputs; the feature representation of the non-text modal inputs is used to replace the feature representation of the corresponding modality identification information in the feature representation of the plain text inputs to obtain the feature representation of the multimodal fusion.

[0205] In an optional embodiment, after encoding the non-textual modal inputs included in the multimodal input data to obtain feature representations of the non-textual modal inputs, the multimodal task processing module 92 is further configured to:

[0206] A sequence of feature representations of at least one input of the same non-text modality in the multimodal input data is stored in correspondence with the modality identification information of the non-text modality.

[0207] In an optional embodiment, when replacing non-text modal inputs in multimodal input data with corresponding modality identification information to obtain plain text input, the multimodal task processing module 92 is further configured to:

[0208] The non-text modal inputs in the multimodal input data are replaced with text fragments of the same length, which are concatenated using the corresponding modal identifier information, to obtain the plain text input.

[0209] In an optional embodiment, when implementing the replacement of the feature representation corresponding to the modality identification information in the feature representation of plain text input with the feature representation of non-text modal input to obtain a multimodal fusion feature representation, the multimodal task processing module 92 is further configured to:

[0210] For any feature representation of a non-text modal input, based on the length of the feature representation of the non-text modal input, a text fragment of the same length, formed by concatenating the corresponding modal identifier information, is found in the plain text input to determine the position of the text fragment in the plain text input; based on the position of the text fragment in the plain text input, the content at the corresponding position in the feature representation of the plain text input is replaced with the feature representation of the non-text modal input to obtain the multimodal fusion feature representation.

[0211] In an optional embodiment, the multimodal input data to be processed includes multimodal input data of multiple task requests, which constitute a set of requests. When encoding the plain text input to obtain a feature representation of the plain text input, the multimodal task processing module 92 is further configured to:

[0212] The plain text input of each task request is passed to the preset encoding interface. The maximum length of the plain text input of each task request is set through the preset encoding interface. By filling the end of the plain text input with placeholders that are shorter than the maximum length, the lengths of the plain text input of multiple task requests are aligned. The aligned plain text input of multiple task requests is then encoded.

[0213] Furthermore, when encoding aligned plain text input from multiple task requests, the multimodal task processing module 92 is also used to:

[0214] The system parses the placeholders contained in the aligned plain text input to determine the actual effective length of the aligned plain text input and the placeholder offsets of the contained placeholders. Based on the actual effective length of the aligned plain text input and the placeholder offsets of the contained placeholders, the system encodes the actual effective input from the aligned plain text input of multiple task requests.

[0215] In an optional embodiment, the model optimization module 93 is specifically used to: before inputting multimodal input data into the target task model, parse the initial task model to obtain operators of multiple model units in the initial task model; construct a model representation graph corresponding to the initial task model with each operator as a node; identify the operator information of each operator in the model representation graph; optimize the computational efficiency of each operator using the computational efficiency improvement strategy corresponding to the operator information to obtain the target operator; and construct the target task model based on the target operator.

[0216] The apparatus provided in this application embodiment can be specifically used to execute the processing flow executed by the server in any of the above method embodiments. The specific functions and technical effects achieved will not be elaborated here.

[0217] Figure 10 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Figure 10As shown, the server includes a memory 1001 and a processor 1002. The memory 1001 stores computer-executable instructions and can be configured to store various other data to support operations on the server. The processor 1002 is communicatively connected to the memory 1001 and executes the computer-executable instructions stored in the memory 1001 to implement the technical solutions provided in any of the above method embodiments. Their specific functions and the technical effects they achieve are similar and will not be repeated here. Figure 10 The example provided uses a cloud server deployed in the cloud as an illustration. However, a server can also be a server deployed locally, without any specific limitation.

[0218] Optional, such as Figure 10 As shown, the server also includes other components such as a firewall 1003, a load balancer 1004, a communication component 1005, and a power supply component 1006. Figure 10 The diagram only shows some components and does not mean that the server only includes... Figure 10 The components shown.

[0219] This application also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, the computer-executable instructions are used to implement the technical solutions provided in any of the above method embodiments. The specific functions and technical effects to be achieved are not described here.

[0220] This application also provides a computer program product, which includes a computer program stored in a readable storage medium. At least one processor of the server can read the computer program from the readable storage medium, and the at least one processor executes the computer program to cause the server to perform the technical solution provided in any of the above method embodiments. The specific functions and technical effects that can be achieved are not described here.

[0221] This application provides a chip, including a processing module and a communication interface. The processing module is capable of executing the technical solution of the server in the aforementioned method embodiments. Optionally, the chip further includes a storage module (e.g., a memory), which stores instructions. The processing module executes the instructions stored in the storage module, and the execution of the instructions stored in the storage module causes the processing module to execute the technical solution provided in any of the aforementioned method embodiments.

[0222] The aforementioned memory can be object storage (OSS). This memory can be implemented using any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0223] The aforementioned communication components are configured to facilitate wired or wireless communication between the device housing the communication components and other devices. The device housing the communication components can access wireless networks based on communication standards, such as mobile hotspots (WiFi), second-generation (2G), third-generation (3G), fourth-generation (4G) / Long Term Evolution (LTE), fifth-generation (5G), or combinations thereof. In one exemplary embodiment, the communication components receive broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication components also include a Near Field Communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID), Infrared Data Association (IrDA) technology, Ultra-Wideband (UWB) technology, Bluetooth (BT) technology, and other technologies. The aforementioned power supply components provide power to various components of the device housing the power supply components. The power supply components may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the device housing the power supply components.

[0224] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, read-only optical disc storage (CD-ROM), optical storage, etc.) containing computer-usable program code.

[0225] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0226] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0227] In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, a network interface, and memory. Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0228] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0229] It should be noted that the user information (including but not limited to user device information, user attribute information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with relevant laws, regulations and standards, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0230] Furthermore, in some of the processes described in the above embodiments and accompanying drawings, multiple operations appear in a specific order. However, it should be clearly understood that these operations may not be executed in the order they appear herein, or may be executed in parallel. The sequence numbers are merely used to distinguish different operations, and the sequence number itself does not represent any execution order. Additionally, these processes may include more or fewer operations, and these operations may be executed sequentially or in parallel. It should be noted that the descriptions such as "first," "second," etc., in this document are used to distinguish different messages, devices, modules, etc., and do not represent a sequential order, nor do they limit "first" and "second" to different types. "Multiple" means two or more, unless otherwise explicitly specified.

[0231] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0232] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A multimodal task processing method, characterized in that, include: Acquire the multimodal input data to be processed; The multimodal input data is input into the target task model, which encodes and fuses the multimodal input data to obtain a multimodal fused feature representation. Task processing is then performed based on the multimodal fused feature representation to generate task processing results. The target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy. The process of encoding and fusing the multimodal input data to obtain a multimodal fused feature representation includes: The non-textual modal inputs included in the multimodal input data are encoded to obtain the feature representations of the non-textual modal inputs; The non-text modal inputs in the multimodal input data are replaced with the corresponding modal identifier information to obtain plain text input; The plain text input is encoded to obtain a feature representation of the plain text input; The feature representation of the non-text modal input is used to replace the feature representation of the corresponding modality identification information in the feature representation of the plain text input to obtain the feature representation of multimodal fusion.

2. The method according to claim 1, characterized in that, After encoding the non-textual modal inputs included in the multimodal input data to obtain the feature representations of the non-textual modal inputs, the method further includes: The sequence of feature representations of at least one input of the same non-text modality in the multimodal input data is stored in correspondence with the modality identification information of the non-text modality.

3. The method according to claim 1, characterized in that, The step of replacing non-text modal inputs in the multimodal input data with corresponding modal identifier information to obtain plain text input includes: The non-text modal inputs in the multimodal input data are replaced with text fragments of the same length, which are concatenated using the corresponding modal identifier information, to obtain plain text input.

4. The method according to claim 3, characterized in that, The step of replacing the feature representation of the corresponding modality identification information in the feature representation of the plain text input with the feature representation of the non-text modality input to obtain a multimodal fusion feature representation includes: For any of the feature representations of the non-text modal input, based on the length of the feature representation of the non-text modal input, a text fragment of the same length, formed by splicing the corresponding modal identifier information, is found in the plain text input to determine the position of the text fragment in the plain text input; Based on the position of the text fragment in the plain text input, the content at the corresponding position in the feature representation of the plain text input is replaced using the feature representation of the non-text modal input to obtain a multimodal fusion feature representation.

5. The method according to any one of claims 1-4, characterized in that, The multimodal input data to be processed includes multimodal input data of multiple task requests, and the multiple task requests constitute a set of requests; Encoding the plain text input to obtain a feature representation of the plain text input includes: The plain text input of each task request is passed to a preset encoding interface. The maximum length of the plain text input of each task request is set through the preset encoding interface. By filling the end of the plain text input with placeholders whose length is less than the maximum length, the lengths of the plain text input of the multiple task requests are aligned. The aligned plain text input of the multiple task requests is then encoded.

6. The method according to claim 5, characterized in that, Encoding the aligned plain text input of the multiple task requests includes: The placeholders contained in the aligned plain text input are parsed to determine the actual effective length of the aligned plain text input and the placeholder offset of the contained placeholders. Encode the actual valid input from the aligned plain text input of multiple task requests based on the actual valid length of the aligned plain text input and the placeholder offset of the included placeholders.

7. The method according to any one of claims 1-4, characterized in that, Before inputting the multimodal input data into the target task model, the method further includes: The initial task model is parsed to obtain operators for multiple model units in the initial task model; Using each operator as a node, construct the model representation graph corresponding to the initial task model; Identify the operator information of each operator in the model representation graph; By utilizing the computational efficiency improvement strategies corresponding to the operator information, the computational efficiency of each operator is optimized to obtain the target operator; Based on the target operator, construct the target task model.

8. A method for processing multimodal dialogue tasks, characterized in that, Applied to servers, including: A dialogue task request sent by a receiving end-side device, the dialogue request containing multimodal input data; The multimodal input data is input into the target dialogue model. The target dialogue model encodes and fuses the multimodal input data to obtain a multimodal fused feature representation. Dialogue task processing is then performed based on the multimodal fused feature representation to generate response text. The target dialogue model is obtained by optimizing the computational efficiency of the operators of the model units in the initial dialogue model using an optimization strategy. The response text is sent to the terminal device; The process of encoding and fusing the multimodal input data to obtain a multimodal fused feature representation includes: The non-textual modal inputs included in the multimodal input data are encoded to obtain the feature representations of the non-textual modal inputs; The non-text modal inputs in the multimodal input data are replaced with the corresponding modal identifier information to obtain plain text input; The plain text input is encoded to obtain a feature representation of the plain text input; The feature representation of the non-text modal input is used to replace the feature representation of the corresponding modality identification information in the feature representation of the plain text input to obtain the feature representation of multimodal fusion.

9. A multimodal dialogue task processing system, characterized in that, Including end-side devices and servers, The endpoint device is used to receive a dialogue task request sent by the endpoint device and send multimodal input data to the server. The dialogue task request includes multimodal input data. The server is used to receive multimodal input data, input the multimodal input data into a target dialogue model, encode and fuse the multimodal input data through the target dialogue model to obtain a multimodal fused feature representation, and perform dialogue task processing based on the multimodal fused feature representation to generate response text; wherein, the target dialogue model is obtained by optimizing the computational efficiency of the operators of the model units in the initial dialogue model using an optimization strategy; The server is also used to send the response text to the end-side device; The end-side device is also used to receive and output the response text; The server is specifically used to encode the non-text modal inputs included in the multimodal input data to obtain the feature representation of the non-text modal input; replace the non-text modal inputs in the multimodal input data with the corresponding modality identification information to obtain plain text input; encode the plain text input to obtain the feature representation of the plain text input; and replace the feature representation of the corresponding modality identification information in the feature representation of the plain text input with the feature representation of the non-text modal input to obtain the feature representation of multimodal fusion.

10. A multimodal task processing system, characterized in that, Including end-side devices and servers, The end-side device is used to receive multimodal input data to be processed from the user and send the multimodal input data to the server; The server is used to receive multimodal input data sent by the end-side device, input the multimodal input data into the target task model, encode and fuse the multimodal input data through the target task model to obtain a multimodal fused feature representation, and perform task processing based on the multimodal fused feature representation to generate task processing results; wherein, the target task model is obtained by optimizing the computational efficiency of the operators of the model units in the initial task model using an optimization strategy; The server is also used to send the task processing result to the end-side device; The end-side device is also used to receive task processing results and output the task processing results; The server is specifically used to encode the non-text modal inputs included in the multimodal input data to obtain the feature representation of the non-text modal input; replace the non-text modal inputs in the multimodal input data with the corresponding modality identification information to obtain plain text input; encode the plain text input to obtain the feature representation of the plain text input; and replace the feature representation of the corresponding modality identification information in the feature representation of the plain text input with the feature representation of the non-text modal input to obtain the feature representation of multimodal fusion.

11. A server, characterized in that, include: A processor, and a memory communicatively connected to the processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory to implement the method as described in any one of claims 1-8.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-8.

Images