Multi-modal data processing method and device and model training method and device

Abstract

The embodiment of the invention provides a multi-modal data processing method and device and a model training method and device.According to the multi-modal data processing method, to-be-processed data are input into a multi-modal data processing model, so that the multi-modal data processing model efficiently outputs target data after being processed in an inference stage. Specifically, in a reasoning stage, for a target model layer in a plurality of model layers, sparse processing is carried out on a to-be-calculated first modal feature unit set by using a first modal sparse predictor to obtain a first modal sparse feature unit set, and sparse processing is carried out on the first modal feature unit set by using the first modal sparse predictor to obtain a second modal sparse feature unit set; the number of feature units participating in calculation is effectively reduced, and the subsequent calculation process is accelerated. And secondly, the first modal sparse feature unit set subjected to sparse processing is input into the target model layer for calculation, so that the burden of the multi-modal data processing model can be reduced, and the multi-modal data processing model is more efficient when processing complex tasks.

Description

Technical Field

[0001] The embodiments of this specification relate to the field of artificial intelligence technology, and in particular to a multimodal data processing method and device, and a model training method and device. Background Art

[0002] Existing large multimodal language models usually use a Transformer architecture with only a decoder, which has a very large parameter scale, enabling it to efficiently handle complex multimodal information fusion tasks (such as text and visual information fusion tasks). However, as the output sequence grows, the computational burden of the large multimodal language model gradually increases, resulting in significant consumption of computing resources.

[0003] In order to improve the efficiency of reasoning, KV cache (Key-Value cache) technology is widely used. This technology reduces redundant calculations by storing and reusing previously calculated key-value pair activation values, thereby improving reasoning efficiency. However, this also brings about the problem of increased GPU (Graphics Processing Unit) memory overhead, especially when long sequences are generated, the memory usage is particularly obvious.

[0004] Therefore, how to improve the reasoning efficiency of large multimodal language models without increasing computing resources and GPU memory usage has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] In view of this, an embodiment of this specification provides a multimodal data processing method. One or more embodiments of this specification also relate to a multimodal data processing device, a model training method, a model training device, a computing device, a computer-readable storage medium, and a computer program product to solve the technical defects existing in the prior art.

[0006] According to a first aspect of an embodiment of this specification, a multimodal data processing method is provided, including: Inputting the data to be processed into the multimodal data processing model so that the multimodal data processing model outputs target data after processing in the inference phase, wherein the data to be processed includes at least first modality data, the target data includes at least second modality data, and the multimodal data processing model includes a plurality of model layers, each model layer performing corresponding calculations in a corresponding inference phase based on a set of feature units; In the inference stage, for a target model layer among the multiple model layers, a first modal feature unit set to be calculated is sparsely processed by using a first modal sparse predictor to obtain a first modal sparse feature unit set, wherein the number of feature units in the first modal sparse feature unit set is less than the number of feature units in the first modal feature unit set; The first modal sparse feature unit set is input into the target model layer so that the target model layer performs calculation based on the first modal sparse feature unit set.

[0007] According to a second aspect of an embodiment of this specification, a multimodal data processing device is provided, including: A data processing module is configured to input the data to be processed into the multimodal data processing model, so that the multimodal data processing model outputs target data after processing in the inference phase, wherein the data to be processed includes at least first modality data, the target data includes at least second modality data, and the multimodal data processing model includes a plurality of model layers, each model layer performs corresponding calculations in a corresponding inference phase based on a set of feature units; A first sparse module is configured to, in the inference stage, for a target model layer among the multiple model layers, perform sparse processing on the first modal feature unit set to be calculated by using a first modal sparse predictor to obtain a first modal sparse feature unit set, wherein the number of feature units in the first modal sparse feature unit set is less than the number of feature units in the first modal feature unit set; The data calculation module is configured to input the first modal sparse feature unit set into the target model layer so that the target model layer performs calculation based on the first modal sparse feature unit set.

[0008] According to a third aspect of an embodiment of this specification, a model training method is provided, including: Acquire a training sample data set, wherein the training sample data set includes a plurality of sample data to be processed, and the sample data to be processed includes at least first modality data; The sample data to be processed is input into a multimodal data processing model, so that the multimodal data processing model outputs predicted sample data after being processed in a forward propagation phase in a training phase, wherein the predicted sample data at least includes second modality data, and the multimodal data processing model includes a plurality of model layers, each model layer performs corresponding calculations in a corresponding training phase based on a set of feature units, In the forward propagation phase of the training phase, for a target model layer among the multiple model layers, a first modal feature unit set to be calculated is sparsely processed by using a first modal sparse predictor to obtain a first modal sparse feature unit set, wherein the number of feature units in the first modal sparse feature unit set is less than the number of feature units in the first modal feature unit set; Inputting the first modal sparse feature unit set into the target model layer so that the target model is calculated based on the first modal sparse feature unit set; In the back propagation stage of the training stage, a target loss value is calculated based on the predicted sample data and the preset sample data, and the multimodal data processing model and the first modality sparse predictor are updated and trained based on the target loss value.

[0009] According to a fourth aspect of the embodiments of this specification, a model training device is provided, including: A data acquisition module is configured to acquire a training sample data set, wherein the training sample data set includes a plurality of sample data to be processed, and the sample data to be processed includes at least first modality data; The data processing module is configured to input the sample data to be processed into the multimodal data processing model, so that the multimodal data processing model outputs predicted sample data after processing in the forward propagation phase in the training phase, wherein the predicted sample data at least includes the second modality data, and the multimodal data processing model includes a plurality of model layers, each model layer performs corresponding calculations in a corresponding training phase based on a set of feature units, A first sparse module is configured to perform sparse processing on a first modal feature unit set to be calculated using a first modal sparse predictor for a target model layer in the multiple model layers during a forward propagation phase in the training phase, to obtain a first modal sparse feature unit set, wherein the number of feature units in the first modal sparse feature unit set is less than the number of feature units in the first modal feature unit set, a data calculation module, configured to input the first modal sparse feature unit set into the target model layer so that the target model performs calculation based on the first modal sparse feature unit set, The training module is configured to calculate a target loss value according to the predicted sample data and the preset sample data in the back propagation stage of the training stage, and update the training of the multimodal data processing model and the first modality sparse predictor according to the target loss value.

[0010] According to a fifth aspect of an embodiment of this specification, a computing device is provided, including: Memory and processor; Among them, the memory is used to store computer programs / instructions, and the processor is used to execute computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the above-mentioned multimodal data processing method or model training method are implemented.

[0011] According to the sixth aspect of the embodiments of this specification, a computer-readable storage medium is provided, which stores a computer program / instruction, and when the computer program / instruction is executed by a processor, the steps of the above-mentioned multimodal data processing method or model training method are implemented.

[0012] According to the seventh aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instruction, which implements the steps of the above-mentioned multimodal data processing method or model training method when executed by a processor.

[0013] An embodiment of the present specification implements a multimodal data processing method, inputting the data to be processed into the multimodal data processing model so that the multimodal data processing model can efficiently output the target data after being processed in the reasoning stage. Specifically, in the reasoning stage, for the target model layer in the multiple model layers, the first modal feature unit set to be calculated is sparsely processed using the first modal sparse predictor to obtain the first modal sparse feature unit set, and the first modal feature unit set is sparsely processed by using the first modal sparse predictor to effectively reduce the number of feature units involved in the calculation, accelerate the subsequent calculation process, and greatly improve the reasoning efficiency of the multimodal data processing model without increasing the computing resources and GPU memory usage. And because the reduced amount of data is generally composed of redundant or irrelevant information, the implementation process can also improve the focus of the multimodal data processing model for data processing, making the key information more prominent, and greatly improving the accuracy of the multimodal data processing model in the reasoning stage. Secondly, the sparsely processed first modal sparse feature unit set is input into the target model layer for calculation, which can reduce the burden of the multimodal data processing model and make it more efficient when processing complex tasks. By optimizing the data of a specific modality through this multimodal data processing method, the entire multimodal data processing model can better and more efficiently integrate multiple information sources to generate more accurate and rich target data. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 A schematic diagram of a scenario of a multimodal data processing method provided according to an embodiment of this specification is shown; Figure 2 A flowchart of a multimodal data processing method provided according to an embodiment of the present specification is shown; Figure 3 A schematic diagram of the reasoning process of a multimodal data processing model in a multimodal data processing method provided according to an embodiment of the present specification is shown; Figure 4 A method flow chart of a model training method provided according to an embodiment of this specification is shown; Figure 5 A schematic diagram of the structure of an image predictor in a model training method provided according to an embodiment of this specification is shown; Figure 6 A schematic diagram of the structure of a text predictor in a model training method provided according to an embodiment of this specification is shown; Figure 7 A schematic diagram of the structure of a multimodal data processing device provided according to an embodiment of the present specification is shown; Figure 8 A schematic diagram of the structure of a model training device provided according to an embodiment of this specification is shown; Fig. 9 A structural block diagram of a computing device provided according to an embodiment of the present specification is shown. DETAILED DESCRIPTION

[0015] Many specific details are described in the following description to facilitate a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without violating the connotation of this specification, so this specification is not limited to the specific implementation disclosed below.

[0016] The terms used in one or more embodiments of this specification are only for the purpose of describing specific embodiments, and are not intended to limit one or more embodiments of this specification. The singular forms of "a", "said" and "the" used in one or more embodiments of this specification and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0017] It should be understood that although the terms first, second, etc. may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of one or more embodiments of this specification, the first may also be referred to as the second, and similarly, the second may also be referred to as the first. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0018] In addition, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0019] In one or more embodiments of this specification, a large model refers to a deep learning model with large-scale model parameters, which usually contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than 10 trillion model parameters. A large model can also be called a foundation model / foundation model. The large model is pre-trained with large-scale unlabeled corpus to produce a pre-trained model with more than 100 million parameters. This model can adapt to a wide range of downstream tasks, and the model has good generalization ability, such as a large-scale language model (LLM, Large Language Model), a multi-modal pre-training model, etc.

[0020] When the big model is used in practice, only a small number of samples are needed to fine-tune the pre-trained model and it can be applied to different tasks. The big model can be widely used in natural language processing (NLP), computer vision and other fields. Specifically, it can be applied to computer vision tasks such as visual question answering (VQA), image description (IC, Image Caption), image generation, as well as natural language processing tasks such as text-based sentiment classification, text summary generation, and machine translation. The main application scenarios of the big model include digital assistants, intelligent robots, search, online education, office software, e-commerce, intelligent design, etc.

[0021] First, the terms involved in one or more embodiments of this specification are explained.

[0022] Transformer: A deep learning model architecture that relies entirely on attention to capture global dependencies in a sequence.

[0023] KV Cache: Key-Value Cache is an optimization technology used by the Transformer model in the autoregressive decoding process. It aims to reduce repeated calculations by caching previously calculated key (Key) and value (Value) vectors, thereby improving reasoning efficiency.

[0024] Token: Token is the basic unit for the model to process modal data (such as text or images), and is referred to as a feature unit in the following text. For text, it refers to the smallest unit after the text is segmented, which can be a word, subword or character. For images, it can be the basic processing unit of the image. Each token will be mapped to a vector (embedded representation) for further processing by the model.

[0025] To solve the above technical problems, in this specification, a multimodal data processing method is provided. This specification also involves a multimodal data processing arrangement, a model training method, a model training device, a computing device, a computer-readable storage medium, and a computer program product, which are described in detail one by one in the following embodiments.

[0026] See also Figure 1 , Figure 1 A schematic diagram of a scenario of a multimodal data processing method provided according to an embodiment of the present specification is shown. In actual applications, the multimodal data processing method is implemented by using the terminal device 102 and the server 104, and the specific implementation is as follows: First, in response to a user trigger, the terminal device 102 sends data to be processed to the server 104, namely, first modal data and a task text corresponding to the first modal data. For example, if the first modal data is an image containing three puppies, the task text is "Please answer how many puppies are there in the image". Wherein, when the first modal data is an image, the user can upload the image collected by a third party to the terminal device 102 by uploading, or can directly collect the image through the terminal device 102 if the terminal device 102 has an image collection function. The embodiments of this specification do not limit this.

[0027] When the server 104 receives the first modal data and the task text corresponding to the first modal data sent by the terminal device 102, the multimodal data processing method is applied to perform the following processing: Inputting the data to be processed into the multimodal data processing model so that the multimodal data processing model outputs target data after processing in the inference phase, wherein the data to be processed includes at least first modality data, the target data includes at least second modality data, and the multimodal data processing model includes a plurality of model layers, each model layer performing corresponding calculations in a corresponding inference phase based on a set of feature units; In the inference stage, for a target model layer among the multiple model layers, a first modal feature unit set to be calculated is sparsely processed by using a first modal sparse predictor to obtain a first modal sparse feature unit set, wherein the number of feature units in the first modal sparse feature unit set is less than the number of feature units in the first modal feature unit set; The first modal sparse feature unit set is input into the target model layer so that the target model layer performs calculation based on the first modal sparse feature unit set.

[0028] Among them, the multimodal data processing model and the first modal sparse predictor can be understood as the multimodal data processing model and the first modal sparse predictor that are updated and trained according to the training samples in the sample library.

[0029] Still taking the example where the first modal data is an image containing three puppies and the task text is "Please answer how many puppies are there in the image", after the multimodal data processing method performs data processing, the multimodal data processing model outputs the second modal data after processing in the reasoning stage, which can be understood as: according to the task text "Please answer how many puppies are there in the image", the text description corresponding to "the image containing three puppies" is generated: "There are three puppies in the image".

[0030] After obtaining the second modality data, the server 104 may return the second modality data to the terminal device 102 .

[0031] The end-side device 102 may include a browser, an APP (Application), or a web application such as an H5 (Hyper Text Markup Language 5, version 5 of Hypertext Markup Language) application, or a light application (also known as a mini-program, a lightweight application) or a cloud application, etc. The end-side device may be based on a software development kit (SDK) of the corresponding service provided by the server, such as based on the real-time communication (RTC, Real Time Communication) SDK development and acquisition. The end-side device may be deployed in an electronic device and needs to rely on the device to run or some APPs in the device to run. The electronic device may have a display screen and support information browsing, such as a personal mobile terminal such as a mobile phone, a tablet computer, a personal computer, etc. Various other types of applications may also be configured in the electronic device, such as human-computer dialogue applications, model training applications, text processing applications, web browser applications, shopping applications, search applications, instant messaging tools, email clients, social platform software, etc.

[0032] Server 104 can be understood as a server that provides various services, including physical servers and cloud servers, such as a server that provides communication services to multiple clients, a server for background training that supports the model used on the client, and a server that processes the data sent by the client. It should be noted that server 104 can be implemented as a distributed server cluster consisting of multiple servers, or as a single server. Server 104 can also be a server for a distributed system, or a server combined with a blockchain. Server 104 can also be a cloud server for basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (CDNs), and big data and artificial intelligence platforms, or an intelligent cloud computing server or intelligent cloud host with artificial intelligence technology.

[0033] In the actual application scenarios of content application platforms, multimodal large language models can process various types of data including images, text, videos, etc., and generate representations (embeddings) with rich semantic information. These representations can be widely used in downstream tasks such as search and recommendation.

[0034] Specifically, in terms of search, a multimodal large language model can be used to obtain a representation of notes (which may include data in one or more modalities) on a content application platform, and then based on the representation of the notes, find notes that match the query terms entered by the user.

[0035] In terms of recommendation, based on the user's historical behavior (browsing, liking, commenting, collecting, etc.), combined with the representation of the notes, more personalized recommended content is provided to the user. For example, a multimodal large language model can be used to obtain the representation of the notes on the content application platform, and then based on the representation of the notes, the notes that match the user (for example, matching the user's historical behavior and / or interests, etc.) can be found to recommend to the user.

[0036] The multimodal data processing method provided by an embodiment of the present specification inputs the data to be processed into the multimodal data processing model so that the multimodal data processing model can efficiently output the target data after being processed in the reasoning stage. Specifically, in the reasoning stage, for the target model layer in the multiple model layers, the first modal feature unit set to be calculated is sparsely processed using the first modal sparse predictor to obtain the first modal sparse feature unit set, and the first modal feature unit set is sparsely processed by using the first modal sparse predictor to effectively reduce the number of feature units involved in the calculation, accelerate the subsequent calculation process, and greatly improve the reasoning efficiency of the multimodal data processing model without increasing the computing resources and GPU memory usage. And because the reduced amount of data is generally composed of redundant or irrelevant information, the implementation process can also improve the focus of the multimodal data processing model for data processing, making the key information more prominent, and greatly improving the accuracy of the multimodal data processing model in the reasoning stage. Secondly, the sparsely processed first modal sparse feature unit set is input into the target model layer for calculation, which can reduce the burden of the multimodal data processing model and make it more efficient when processing complex tasks. By optimizing the data of a specific modality through this multimodal data processing method, the entire multimodal data processing model can better and more efficiently integrate multiple information sources to generate more accurate and rich target data.

[0037] See also Figure 2 , Figure 2 A flowchart of a multimodal data processing method provided according to an embodiment of the present specification is shown, which specifically includes the following steps.

[0038] Step 202: Input the data to be processed into the multimodal data processing model so that the multimodal data processing model outputs the target data after processing in the inference phase.

[0039] The data to be processed includes at least first modal data, the target data includes at least second modal data, and the multimodal data processing model includes multiple model layers, each model layer performs corresponding calculations in a corresponding reasoning stage based on a set of feature units.

[0040] Step 204: In the inference stage, for the target model layer among the multiple model layers, the first modal feature unit set to be calculated is sparsely processed by using the first modal sparse predictor to obtain the first modal sparse feature unit set.

[0041] The number of feature units in the first modal sparse feature unit set is less than the number of feature units in the first modal feature unit set.

[0042] Step 206: Input the first modal sparse feature unit set into the target model layer so that the target model layer performs calculations based on the first modal sparse feature unit set.

[0043] Among them, the multiple model layers of the multimodal data processing model include but are not limited to input layer, encoding layer, fusion layer, attention layer, decoding layer, output layer, etc.; in practical applications, the target model layer can be any layer in the multimodal data processing model, and the model layer involving token sparse processing in the processing process can be understood as the target model layer.

[0044] Specifically, the first modality data can be determined according to the actual data processing task. For example, in the case where the data processing task is to generate text from an image, the first modality data can be understood as image data, and the second modality data can be understood as text data, that is, the first modality includes images and the second modality includes text; and in the case where the data processing task is to generate images from text, the first modality data can be understood as text data, and the second modality data can be understood as image data.

[0045] Then, in the case where the data processing task of the first modal data is a data processing task of generating text from an image, the task text can be understood as a text used to describe the generation of a corresponding text from an image. For example, if the first modal data is “an image of three puppies running on the grass” and the data processing task of the first modal data is a data processing task of “generating text based on the image”, the task text can be “please describe the image content of the image through text”; and in the case where the data processing task of the first modal data is a data processing task of generating an image from a text, the task text can be understood as a text used to describe the generation of a corresponding image based on a text. For example, if the first modal data is “text of three puppies running on the grass” and the data processing task of the first modal data is a data processing task of “generating an image based on the text”, the task text can be “please generate a corresponding image based on the text”.

[0046] For ease of understanding, the following embodiments are all described in detail by taking the first modality as an image and the second modality as text, that is, the data processing task is a data processing task of generating text from an image, and the task text is a text used to describe the generation of corresponding text from an image as an example.

[0047] In specific implementation, the multimodal data processing model can be understood as a multimodal large language model, and the multimodal data processing model can be understood as a multimodal data processing model obtained by updating training in the following model training method. In the process of graph-to-text reasoning, the multimodal large language model includes a prefill stage and a decoding stage. The prefill stage can be expressed by Formula 1: Formula 1 in, , Indicates participation in the multimodal large language model The image token set calculated by the layer (i.e. The first modal feature unit set of the layer output), Indicates participation in the multimodal large language model The text token set calculated by the layer, and They represent the participation in the multimodal large language model. Layer and The image token and text token set calculated by the layer, and They respectively represent the multi-head attention layer and feed-forward network layer of the multimodal large language model.

[0048] The decoding stage can be expressed by formula 2: Formula 2 in, Indicates participation in the multimodal large language model Layer calculation, output text token set (i.e. The second modal feature unit set of the layer output), U represents the union of the sets.

[0049] According to Formula 1 and Formula 2, it can be understood that the pre-filling stage and the decoding stage constitute the reasoning process of the multimodal large language model, that is, the reasoning stage of the multimodal large language model includes the pre-filling stage and the decoding stage, wherein the pre-filling stage is used to initialize and fuse information from different modalities (such as images and texts), and the decoding stage can gradually generate new output texts on this basis; wherein, in the case where the first modality data is image data, the first modality feature unit set can be understood as the image token set in the above formula 1, and the text features of the above task text can be understood as the text token set in the above formula 1, and the first modality feature unit set and the text features of the task text can be understood as the image token set and the text token set obtained for use in the decoding stage after being processed by the multi-attention layer and the feedforward network layer in the pre-filling stage.

[0050] Then, after obtaining the data to be processed as described above, feature processing can be performed on the data to be processed in the pre-filling stage of the inference stage to obtain a first modal feature unit set for subsequent decoding processing. For example, when the first modal feature unit set is an image token set, feature processing is performed on the data to be processed in the pre-filling stage to obtain a first modal feature unit set. This can be understood as performing attention learning and nonlinear transformation on the image token set through the multi-head attention layer and feedforward network layer of the multimodal data processing model in the pre-filling stage, from the preliminary image token set extraction to the learning of complex semantics and logical relationships, to obtain the final image token set (i.e., the first modal feature unit set) that includes basic context perception and consistent dimensions; of course, when it is necessary to perform feature processing on the task text to obtain a text token set, you can also refer to this specific implementation.

[0051] By using the multi-head attention layer and feedforward network layer of the multimodal data processing model to perform attention learning and nonlinear transformation on the data to be processed in the pre-filling stage of the inference phase, complex semantics and logical relationships are extracted from the preliminary data to be processed, helping the multimodal data processing model to capture key information in the input data earlier, providing a better basic representation for the subsequent decoding stage, thereby improving the decoding accuracy of the decoding stage.

[0052] After obtaining the first modal feature unit set, in order to improve the reasoning efficiency of the multimodal data processing model, the reasoning process of the multimodal data processing model in the pre-filling stage can be accelerated by reducing the number of feature units in the first modal feature unit set in the pre-filling stage of the reasoning stage, thereby improving the reasoning efficiency of the overall multimodal data processing model. The specific implementation method is as follows: The reasoning stage includes an input processing stage; In the inference stage, for a target model layer among the multiple model layers, a first modal feature unit set to be calculated is sparsely processed by using a first modal sparse predictor to obtain a first modal sparse feature unit set, including: In the input processing stage, for the target model layer among the multiple model layers, the first modal feature unit set to be calculated is sparsely processed by using the first modal sparse predictor to obtain the first modal sparse feature unit set.

[0053] Among them, the input processing stage can be understood as the pre-filling stage.

[0054] As for the method of reducing the feature units in the first modality feature unit set, the first modality sparse predictor can be used for implementation. When the first modality data is image data, the first modality sparse predictor can be understood as an image predictor. For the pre-training steps and update training methods of the first modality sparse predictor, please refer to the detailed introduction in the following model training method.

[0055] In specific implementation, after obtaining the first modal sparse feature unit set, in order to support efficient and accurate reasoning process, the first modal sparse feature unit set implemented by the first modal sparse predictor will be input into the target model layer of the multimodal data processing model, so that the target model layer can perform in-depth analysis and calculation on the first modal sparse feature unit set in the pre-filling stage, thereby supporting the multimodal data processing model to accurately obtain target data. The specific implementation is as follows: The reasoning stage includes an input processing stage; The step of inputting the first modal sparse feature unit set into the target model layer so that the target model layer performs calculation based on the first modal sparse feature unit set includes: The first modal sparse feature unit set is input into the target model layer so that the target model layer performs a first calculation based on the first modal sparse feature unit set during the input processing stage, wherein the first modal feature unit set is obtained based on the first modal data of the data to be processed.

[0056] In practical applications, since the sparse processing of the first modal feature unit set is implemented in the first modal sparse predictor, in order to achieve a higher level of data understanding and output of target data, this requires in-depth analysis and calculation of the multimodal data processing model in the reasoning stage. Therefore, it is necessary to input the first modal sparse feature unit set into the target model layer of the multimodal data processing model so that the target model layer can perform more efficient and accurate calculations based on the first modal sparse feature unit set in the pre-filling stage, so that after a series of processing, the multimodal data processing model outputs accurate target data.

[0057] During the pre-filling stage, the target model layer performs a first calculation based on the first modal sparse feature unit set, including but not limited to feature enhancement, context embedding, attention mechanism, fusion and other specific calculation implementations on the first modal sparse feature unit set, thereby supporting efficient and accurate reasoning process of the multimodal data processing model.

[0058] In one or more embodiments of the present specification, in order to further improve the inference efficiency of the multimodal data processing model, the second modal feature unit set may also be sparsely processed in the inference stage, so as to better improve the inference efficiency of the multimodal data processing model in the inference stage by simultaneously reducing the number of feature units of the first modal feature unit set and the second modal feature unit set in the inference stage. The specific implementation method is as follows: The method further comprises: For the target model layer, the second modality feature unit set to be calculated is sparsely processed by using a second modality sparse predictor to obtain a second modality sparse feature unit set; inputting the second modality sparse feature unit set into the target model layer so that the target model layer performs calculation based on the second modality sparse feature unit set, The number of feature units in the second modality sparse feature unit set is less than the number of feature units in the second modality feature unit set.

[0059] The method of reducing feature units in the second modality feature unit set can be implemented by using a second modality sparse predictor. When the second modality data is text data, the second modality sparse predictor can be understood as a text predictor. For the pre-training steps and update training methods of the second modality sparse predictor, please refer to the detailed introduction in the following model training method.

[0060] Specifically, the first modal sparse feature unit set is input into the target model layer, so that after the target model layer performs a first calculation based on the first modal sparse feature unit set in the pre-filling stage, the first calculation result is decoded in the decoding stage to obtain the second modal feature unit set; in order to improve the reasoning efficiency of the multimodal data processing model in the decoding stage, after obtaining the second modal feature unit set, the target model layer can use a second modal sparse predictor to perform sparse processing on the second modal feature unit set, reduce the number of feature units in the second modal feature unit set, and obtain the second modal sparse feature unit set; so that the subsequent target model layer can perform calculations based on the second modal sparse feature unit set.

[0061] The multimodal data processing method provided by one or more embodiments of the present specification can use a first modality sparse predictor and a second modality sparse predictor to respectively generate sparse decisions for generating a first modality feature unit set and a second modality feature unit set. The sparse first modality sparse feature unit set and the second modality sparse feature unit set with a smaller number of feature units participate in the reasoning process of the multimodal data processing model, thereby greatly accelerating its reasoning speed.

[0062] In practical applications, the decoding process is performed at the decoding stage in the inference stage of the multimodal data processing model. Therefore, the second modal sparse feature unit set is input into the target model layer, and the specific implementation of the calculation based on the second modal sparse feature unit set at the target model layer can be understood as being performed at the decoding stage in the inference stage of the multimodal data processing model. The specific implementation is as follows: The reasoning stage includes an output processing stage; The step of inputting the second modal sparse feature unit set into the target model layer so that the target model layer performs calculation based on the second modal sparse feature unit set includes: Inputting the second modal sparse feature unit set into the target model layer so that the target model layer performs a second calculation based on the second modal sparse feature unit set in the output processing stage, wherein the second modal feature unit set is obtained based on the second modal data of the target data; and / or The step of inputting the first modal sparse feature unit set into the target model layer so that the target model layer performs calculation based on the first modal sparse feature unit set includes: The first modal sparse feature unit set is input into the target model layer so that the target model layer performs a second calculation based on the first modal sparse feature unit set in the output processing stage.

[0063] Similarly, corresponding to the above-mentioned input processing stage being the pre-filling stage, the output processing stage can be understood as the decoding stage.

[0064] Specifically, when the inference stage of the multimodal data processing model is divided into a pre-filling stage and a decoding stage, in the pre-filling stage of the multimodal data processing model, a first modal feature unit set is obtained; then, the first modal feature unit set is sparsely processed using a first modal sparse predictor in the target model layer to obtain a first modal sparse feature unit set; the first modal sparse feature unit set is then input into the target model layer, and in the decoding stage of the multimodal data processing model, the first modal sparse feature unit set is decoded through a decoding network to obtain a second modal feature unit set; on the basis of sparsely processing the second modal feature unit set using a second modal sparse predictor to obtain a second modal sparse feature unit set, the second modal sparse feature unit set is input into the target model layer, so that the target model layer performs a second calculation on the second modal sparse feature unit set in the decoding stage, so as to support the multimodal data processing model to efficiently obtain target data in the inference stage based on the second calculation result.

[0065] During the decoding stage, after the first modal sparse feature unit set and / or the second modal sparse feature unit set are input, the target model layer performs a second calculation based on the first modal sparse feature unit set and / or the second modal sparse feature unit set, including but not limited to specific calculations such as feature fusion, context modeling, and prediction generation for the first modal sparse feature unit set and / or the second modal sparse feature unit set, thereby supporting efficient and accurate reasoning process of the multimodal data processing model.

[0066] In practical applications, in the decoding stage, the target model layer performs a second calculation based on the first modal sparse feature unit set and / or the second modal sparse feature unit set. This can realize the context processing of the multimodal data processing model, utilize information from multiple modalities to achieve information complementarity, enhance expression ability and robustness, and significantly improve model performance and the output quality of target data.

[0067] One or more embodiments of the present specification provide a multimodal data processing method, in which the target model layer of the multimodal data processing model performs a second calculation based on the first modal sparse feature unit set and / or the second modal sparse feature unit set in the decoding stage. This can also improve the model calculation efficiency, reduce memory usage, etc. in the decoding stage by jointly sparsifying the multiple modal sparse feature unit sets, thereby further accelerating the reasoning efficiency of the multimodal data processing model.

[0068] Using the above example, when the first modality data is image data and the second modality data is text data, in the decoding stage of the multimodal data processing model, in order to ensure the coherence and semantic accuracy of the generated text data, an autoregressive method is usually used to generate text describing the image data. Specifically, the target model layer decodes the first modality sparse feature unit set in the decoding stage, and the execution steps of using the autoregressive method to obtain the second modality sparse feature unit set are as follows: In the decoding stage, the target model layer fuses the first modal sparse feature unit set and the task text feature representation (if any) to obtain a multimodal feature unit. According to the multimodal feature unit and the historical second modal feature unit, predict and obtain the i-th second modal feature unit, where i is a positive integer and i starts from 1; Inputting the i-th second modal feature unit into the second modal sparse predictor for sparse processing to obtain the i-th sparse second modal feature unit; Determine whether the preset decoding end condition is met, If yes, the i-th sparse second modal feature unit is determined as a sparse second modal feature unit, and a second modal sparse feature unit set is constructed according to the sparse second modal feature unit; If not, the i-th second modal feature unit is determined as a historical second modal feature unit, and i is incremented by 1, and the step of predicting the i-th second modal feature unit based on the multimodal feature units and the historical second modal feature units is continued, and the i-th sparse second modal feature unit is determined as a historical sparse second modal feature unit until the preset decoding end condition is met, and the historical sparse second modal feature unit and the i-th sparse second modal feature unit are determined as sparse second modal feature units, and a second modal sparse feature unit set is constructed based on the sparse second modal feature units.

[0069] The preset decoding end condition may be set according to actual application, such as checking whether an end mark is generated, or checking whether the generated length meets a preset length threshold.

[0070] In addition, when the first second modal feature unit is generated, there is no historical second modal feature unit, then the historical second modal feature unit can be understood as 0; and as the loop starts, each time a second modal feature unit is generated, the second modal feature unit will serve as the historical second modal feature unit of the next second modal feature unit; the same applies to the historical sparse second modal feature representation.

[0071] The multimodal data processing method provided by one or more embodiments of the present specification, in the decoding stage of the multimodal data processing model, relies on the previously generated second modal feature unit when predicting the next second modal feature unit at each step through an autoregressive method, thereby ensuring the coherence and semantic correctness of the generated second modal data and improving the accuracy of the generation results of the reasoning process; and in the decoding stage, the output second modal feature unit is sparsely processed, and is sparsed once for each predicted second modal feature unit, which helps to optimize the generation process in real time and improve the reasoning efficiency and quality of the multimodal data processing model.

[0072] In addition, in order to further reduce the computational overhead and memory usage of the multimodal data processing model during the inference process, the KV (Key-Value) cache can be used in the decoding stage of the multimodal data processing model to accelerate the autoregressive generation process. By storing the keys and values ​​in the previously calculated attention mechanism, repeated calculations are avoided, thereby improving efficiency. Especially when processing long sequences or performing multiple iterations, the KV cache can significantly reduce computational overhead.

[0073] See also Figure 3 , Figure 3 A schematic diagram of the reasoning process of a multimodal data processing model in a multimodal data processing method provided in one embodiment of the present specification is shown.

[0074] Figure 3The multimodal data processing model in the inference process includes a pre-filling stage and a decoding stage. Taking the first modal data as image data and the second modal data as text data as an example, the specific implementation of the pre-filling stage and the decoding stage of the multimodal data processing model in the inference process is explained.

[0075] In the pre-filling stage, the multimodal data processing model first processes the image tokens in the image token set (i.e., the first modal feature unit set). The image predictor evaluates the image tokens and makes decisions based on certain criteria to determine which image tokens should participate in subsequent calculations. Only image tokens that are judged as "yes!" will be passed to the next layer (i.e., in the pre-filling stage, the multimodal large language model first layer). The layer will receive the sparse image token for data processing. Specifically, the specific implementation of sparse image tokens in the pre-filling stage is shown in Formula 3: Formula 3 in, represents the decision matrix generated by the image predictor, Indicates participation in the multimodal large language model The number of tokens in the image token set calculated by the layer, Indicated in Sampling is performed in the second dimension, Represents a mask matrix with a value of 0 or 1, Represents a collection of image tokens The index of .

[0076] Specifically, the decision matrix is ​​used to evaluate the importance of each image token, and the mask matrix indicates which image tokens are selected (e.g., a value of 1 indicates selected). The selected image tokens constitute a new image token set. , which is smaller than the original image token set .

[0077] According to Formula 3, the sparse processing using the first modality sparse predictor to obtain the first modality sparse feature unit set includes: The method of performing sparse processing using the first modal sparse predictor to obtain a first modal sparse feature unit set includes: generating a first decision matrix using a first modal sparse predictor; Performing sampling using a decision function on a preset dimension of the first decision matrix to generate a first mask matrix; The first modal feature unit set is screened according to the first mask matrix to obtain a first modal sparse feature unit set.

[0078] The first modality sparse predictor is an image predictor, including a linear network layer, a visual processing network layer, and a multi-layer perceptron network layer. The preset dimension is the second dimension. The first decision matrix is ​​the decision matrix generated by the image predictor. The decision function is Function, the first mask matrix can be understood as the above mask matrix , the first modal feature unit set can be understood as the above-mentioned image token set, and the first modal sparse feature unit set can be understood as the above-mentioned new image token set.

[0079] In one or more embodiments of the present specification, a new image token set is used instead of the original image token set to perform calculations in the pre-filling stage of the multimodal large language model. Since the number of tokens in the new image token set is smaller than the number of tokens in the original image token set, the multimodal large language model only needs to process a small number of image tokens in the pre-filling stage, significantly improving the calculation speed of the pre-filling stage.

[0080] In the decoding stage, the multimodal data processing model also adds a text predictor, which evaluates the text tokens in the output text token set (the output text token can be understood as the text token output by the multimodal data processing model) (i.e., the second modal feature unit set), and makes decisions based on certain criteria to determine which text tokens should participate in subsequent calculations. Only text tokens that are judged as "yes!" will be retained as the final output (i.e., in the decoding stage, the multimodal large language model first The layer predicts text tokens based on the sparse tokens of the input, and sparses the predicted text tokens). Specifically, the specific implementation of sparse text tokens in the decoding stage is shown in Formula 4: Formula 4 in, represents the decision matrix generated by the text predictor, Indicates participation in the multimodal large language model The number of tokens in the output text token set calculated by the layer, Indicated in Sampling is performed in the second dimension, Represents a mask matrix with a value of 0 or 1, Represents the output text token set The index of .

[0081] Specifically, the decision matrix is ​​used to evaluate the importance of each text token, and the mask matrix indicates which text tokens are selected (e.g., a value of 1 indicates that they are selected). The selected text tokens constitute a new text token set. , which is smaller than the original text token set .

[0082] According to Formula 4, the sparse processing using the second modality sparse predictor to obtain the second modality sparse feature unit set includes: The sparse processing is performed using the second modality sparse predictor to obtain a second modality sparse feature unit set, including: generating a second decision matrix using a second modality sparse predictor; Performing sampling using a decision function on a preset dimension of the second decision matrix to generate a second mask matrix; The second modality feature unit set is screened according to the second mask matrix to obtain a second modality sparse feature unit set.

[0083] The second modal sparse predictor is a text predictor, including a linear network layer and a multi-layer perceptron network layer. The preset dimension is the second dimension. The second decision matrix is ​​the decision matrix generated by the text predictor. The decision function is Function, the second mask matrix can be understood as the above mask matrix , the second modal feature unit set can be understood as the above-mentioned text token set, and the second modal sparse feature unit set can be understood as the above-mentioned new text token set.

[0084] In one or more embodiments of the present specification, in the decoding stage, an image predictor is still used to obtain a new image token set, or the new image token set is directly used to participate in the calculation of the decoding stage, and then the new text token set is used to replace the original text token set to perform the calculation of the decoding stage of the multimodal large language model. Since the number of tokens in the new text token set is less than the number of tokens in the original text token set, the multimodal large language model only needs to process a small number of text tokens in the decoding stage, which significantly improves the calculation speed of the decoding stage.

[0085] Then, during the inference process of the multimodal data processing model, the image predictor is used to achieve sparse processing of image tokens in the pre-filling stage, and the text predictor is used to achieve sparse processing of text tokens in the decoding stage. Through a small number of image tokens and text tokens, the inference speed of the multimodal data processing model is greatly improved.

[0086] See also Figure 4 , Figure 4 A method flow chart of a model training method provided by an embodiment of the present specification is shown, which specifically includes the following steps.

[0087] Step 402: Obtain a training sample data set.

[0088] The training sample data set includes a plurality of sample data to be processed, and the sample data to be processed includes at least first modality data.

[0089] Step 404: Input the sample data to be processed into the multimodal data processing model, so that the multimodal data processing model outputs predicted sample data after being processed in the forward propagation phase in the training phase.

[0090] The predicted sample data includes at least second modality data, and the multimodal data processing model includes multiple model layers, each model layer performs corresponding calculations in a corresponding training stage based on a set of feature units.

[0091] Step 406: In the forward propagation stage of the training stage, for the target model layer among the multiple model layers, the first modal feature unit set to be calculated is sparsely processed by using the first modal sparse predictor to obtain the first modal sparse feature unit set.

[0092] wherein the number of feature units in the first modal sparse feature unit set is less than the number of feature units in the first modal feature unit set; Step 408: Input the first modal sparse feature unit set into the target model layer so that the target model is calculated based on the first modal sparse feature unit set.

[0093] Step 410: In the back propagation phase of the training phase, a target loss value is calculated based on the predicted sample data and the preset sample data, and the multimodal data processing model and the first modality sparse predictor are updated and trained based on the target loss value.

[0094] Among them, the first modal data can be understood as data of the same modality as the first modal data in the above-mentioned multimodal data processing method, such as image data; the second modal data can be understood as data of the same modality as the second modal data in the above-mentioned multimodal data processing method, such as text data.

[0095] In practical applications, the training sample data set may contain labeled data to implement supervised training, or may not contain labeled data to implement self-supervised training. Then, when the training sample data set contains labeled data to implement supervised training of the multimodal data processing model, the first modality sparse predictor, and the second modality sparse predictor, the preset sample data includes the target sample data corresponding to the sample data to be processed. When the training sample data set does not contain labeled data to implement self-supervised training of the multimodal data processing model, the first modality sparse predictor, and the second modality sparse predictor, the preset sample data can be understood as the pseudo-label generated by the self-supervised learning task: the first classification label corresponding to the sample data to be processed, the target sample data corresponding to the sample data to be processed, and the second classification label corresponding to the target sample data.

[0096] For ease of understanding, the embodiments of this specification are described in detail using the example of a training sample data set containing labeled data to implement supervised training of a multimodal data processing model, a first modality sparse predictor, and a second modality sparse predictor.

[0097] For example, when the first modality data is image data, the first classification label can be understood as an image classification label, and when the second modality data is text data, the second classification label can be understood as a text classification label.

[0098] In practical applications, in order to reduce the processing time of the multimodal data processing model and improve the model training efficiency, the data in the training sample data set can be pre-extracted for features, and the extracted feature data can be directly input into the multimodal data processing model for processing in the training phase to output the predicted sample data. The specific implementation logic can be found in the above multimodal data processing method, which will not be described in detail here.

[0099] Unlike the inference phase, the multimodal data processing model does not distinguish between the pre-filling phase and the decoding phase during the training phase. It adopts an end-to-end approach and regards the entire training process as a continuous task, which simplifies the design and implementation of the model and reduces unnecessary complexity.

[0100] After obtaining the first modal feature unit set of the first modal data, in order to improve the model training efficiency, the model training process can be accelerated by reducing the number of feature units in the first modal feature unit set, thereby improving the overall model training efficiency.

[0101] As for the reduction method of the feature units in the first modality feature unit set, the first modality sparse predictor can be used for implementation, and when the first modality data is image data, the first modality sparse predictor can be understood as an image predictor. The pre-training steps of the image predictor are as follows: Specifically, the pre-training steps of the image predictor are as follows: Determining sample image features of sample image data and image classification labels of the sample image features; Inputting the sample image features into an initial image predictor to obtain a predicted image classification label; According to the image classification label and the predicted image classification label, an image loss value is calculated, and the model parameters of the initial image predictor are adjusted according to the image loss value until a preset model training end condition is met to obtain a pre-trained image predictor.

[0102] Among them, the image classification label of the sample image feature can be understood as the classification label of whether the sample image feature is retained.

[0103] The pre-training steps of the image predictor are explained in detail: first, a sample image is collected, and the image features of the sample image are extracted using the pre-trained image model to obtain the image token (i.e., the sample image feature), and the image classification label of the image token is set, where the image classification label is used to indicate whether the image token is retained, such as 0 for not retaining and 1 for retaining.

[0104] Then, the image token is input into the initial image predictor to obtain an image prediction result of whether the image token is retained, and the image loss value is calculated according to the image prediction result and the image classification label through a preset loss function, and the model parameters of the initial image predictor are adjusted according to the image loss value until the preset model training end condition is met, and a pre-trained image predictor is obtained, wherein the preset model training end condition can be understood as the model iteration meeting the target iteration number, such as two hundred times, or the sample images are all iterated.

[0105] In actual applications, the initial image predictor predicts two values ​​for each image token. When the second value is greater than the first value, the image prediction result can be retained for the image token. Otherwise, the image prediction result is that the image token is discarded.

[0106] Of course, the image prediction result of each image token by the initial image predictor can also be expressed in other ways, such as the initial image predictor scores each image token in two dimensions, gives the values ​​of the two dimensions, and then obtains the image prediction result of whether each image token is retained according to the values ​​of the two dimensions (such as negative and positive) of each image token; or for complex application scenarios, each image token can be scored in multiple dimensions, such as three dimensions (emotional tendency, information importance, context relevance, etc.), and the image prediction result of whether each image token is retained is obtained according to the scores of multiple dimensions. The specific selection is based on actual needs, and one or more embodiments of this specification do not impose any restrictions.

[0107] In multimodal learning tasks, when combined with text information for joint training, attention weights can be generated for the corresponding image parts based on the text content, thereby indirectly forming a validity mark for image tokens, namely a mask. In another possible implementation, the image predictor can also first obtain the image mask through some means (such as object detection, segmentation, attention mechanism, etc.). These masks identify which parts of the image are important or need attention, and then based on these masks, representative image features can be further extracted, namely image tokens. For example, in some visual-language models, a pre-trained object detector is first used to identify and locate important objects in the image, and a mask is generated for each object. Then, the corresponding features are extracted from the original image as image tokens based on these masks.

[0108] In specific implementation, in order to improve the overall performance of the image predictor, make the model more stable during the training process, and maintain good generalization ability, the image predictor can be divided into modules. The specific implementation method is as follows: The image predictor includes a linear network layer, a visual processing network layer, and a multi-layer perceptron network layer; The step of inputting the sample image features into an initial image predictor to obtain a predicted image classification label comprises: Inputting the sample image features into an initial image predictor; In the linear network layer, linear transformation is performed on the sample image features to obtain the sample image features after the linear transformation, and the sample image features after the linear transformation are input into the visual processing network layer; In the visual processing network layer, performing self-attention processing on the sample image features after the linear transformation, obtaining a sample image feature representation after the self-attention processing, and inputting the sample image feature representation into the multi-layer perceptron network layer; In the multi-layer perceptron network layer, classification prediction is performed on the sample image feature representation to obtain a predicted image classification label of the sample image feature.

[0109] Among them, the linear network layer (Linear Layer) is used to perform linear transformation on the sample image features, such as transforming the sample image features into 512-dimensional sample image features through matrix multiplication, that is, the sample image features after linear transformation.

[0110] The Vision Transformer Block is used in a structure based on the self-attention mechanism, which allows the model to focus on different parts of the image, thereby capturing long-distance dependencies and contextual information, and obtaining the feature representation of the sample image after self-attention processing.

[0111] The multilayer perceptron network layer (Multilayer Perceptron) contains one or more layers of nonlinear activation functions, which are used to perform nonlinear transformations on each sample image feature, so that the model can learn more complex feature representations, thereby achieving accurate classification predictions and obtaining predicted image classification labels for sample image features.

[0112] During specific implementation, the specific structure and implementation of the image predictor and the text predictor in the multimodal data processing method may refer to this embodiment.

[0113] The model training method provided by one or more embodiments of the present specification, when the initial image predictor includes a linear network layer, a visual processing network layer, and a multi-layer perceptron network layer, inputs the sample image features into the linear network layer for linear transformation, and reduces the overfitting of the model through the fewer parameters of the linear network layer; then the sample image features after the linear transformation are input into the visual processing network layer for self-attention processing through the self-attention mechanism, so that the model can learn the interaction between different positions and learn the global information; finally, the sample image feature representation after self-attention processing is input into the multi-layer perceptron network layer, and nonlinear transformation is performed through a nonlinear activation function, so that the model learns more complex feature representations and enhances the expression ability of the model; through the overall implementation of the above three-layer structure, the overall performance of the model is improved.

[0114] See also Figure 5 , Figure 5 A schematic diagram of the structure of an image predictor in a model training method provided by an embodiment of the present specification is shown.

[0115] Figure 5The image predictor in includes a linear network layer (Linear Layer), a visual processing network layer (VisionTransformer Block), and a multilayer perceptron network layer (Multilayer Perceptron).

[0116] Using the above example, the image token is input into the Linear Layer. In the Linear Layer, the input channel number is The feature dimension of the image token input to the Linear Layer is converted to 512 dimensions, that is, the input image token is mapped from the original dimension to a new 512-dimensional space, and the output dimension of the image token is 512 dimensions.

[0117] Then, the output of the Linear Layer is input into the visual processing network layer containing 2 Vision Transformer Blocks, and the 512-dimensional image token is processed based on the self-attention mechanism to obtain the 512-dimensional image token after self-attention processing.

[0118] Finally, the output of the visual processing network layer is input into a 3-layer Multilayer Perceptron. The 512-dimensional image token after self-attention processing is nonlinearly transformed using a nonlinear activation function to convert it into a 2-dimensional output, and the prediction result is obtained based on the 2-dimensional output image token.

[0119] certainly, Figure 5 What is shown is only one implementation structure of the image predictor. In practical applications, different parts can be optimized or improved according to actual needs, such as adjusting the depth or width of the Multilayer Perceptron, or changing the number of Vision Transformer Blocks, etc. There is no limitation here.

[0120] In specific implementation, the specific method of obtaining the first modal feature unit set refers to the specific implementation of the above-mentioned multimodal data processing method. After obtaining the first modal sparse feature unit set by using the first modal sparse predictor, the predicted sample data can be output after the multimodal data processing model processes the first modal sparse feature unit set through the training phase.

[0121] Similarly, in order to further improve the model training efficiency, during the training phase, the feature units in the second modal feature unit set can also be sparsely processed, so as to better improve the training efficiency of the multimodal data processing model during the training phase by simultaneously reducing the number of feature units in the first modal feature unit set and the second modal feature unit set during the training process. The specific implementation method is as follows: The method further comprises: In the forward propagation phase of the training phase, for the target model layer, the second modality feature unit set to be calculated is sparsely processed by using the second modality sparse predictor to obtain the second modality sparse feature unit set, Inputting the second modality sparse feature unit set into the target model layer so that the target model layer performs calculation based on the second modality sparse feature unit set, wherein the number of feature units in the second modality sparse feature unit set is less than the number of feature units in the second modality feature unit set; The back propagation phase in the training phase calculates a target loss value according to the predicted sample data and the preset sample data, and updates and trains the multimodal data processing model and the first modality sparse predictor according to the target loss value, including: In the back propagation stage of the training stage, a target loss value is calculated based on the predicted sample data and the preset sample data, and the multimodal data processing model, the first modality sparse predictor, and the second modality sparse predictor are updated and trained based on the target loss value.

[0122] Continuing with the above example, when the second modality data is text data, the second modality sparse predictor can be understood as a text predictor.

[0123] Then, the pre-training steps of the text predictor are as follows: Determining sample text features of sample text data and text classification labels of the sample text features; Inputting the sample text features into an initial text predictor to obtain a predicted text classification label; According to the text classification label and the predicted text classification label, a text loss value is calculated, and the model parameters of the initial text predictor are adjusted according to the text loss value until a preset model training end condition is met, thereby obtaining a pre-trained text predictor.

[0124] Among them, the text classification label can be understood as the classification label of whether the sample text features are retained.

[0125] Specifically, the training steps of the text predictor are explained in detail, that is, sample text is first collected, the text features of the sample text are extracted using a pre-trained language model, text tokens (i.e., sample text features) are obtained, and text classification labels of the text tokens are set, where the text classification labels are used to indicate whether the text token is retained, such as 0 for not retaining and 1 for retaining.

[0126] Then, the text token is input into the initial text predictor to obtain a text prediction result of whether the text token is retained, and the text loss value is calculated according to the text prediction result and the text classification label through a preset loss function, and the model parameters of the initial text predictor are adjusted according to the text loss value until the preset model training end condition is met to obtain a pre-trained text predictor, wherein the preset model training end condition can be understood as the model iteration meeting the target iteration number, such as two hundred times, or the sample texts are all iterated.

[0127] In actual applications, the initial text predictor predicts two values ​​for each text token. When the second value is greater than the first value, the text prediction result can be retained for the text token. Otherwise, the text prediction result is that the text token is discarded.

[0128] Of course, the text prediction result of each text token by the initial text predictor can also be expressed in other ways, such as the initial text predictor scores each text token in two dimensions, gives the values ​​of the two dimensions, and then obtains the text prediction result of whether each text token is retained according to the values ​​of the two dimensions (such as negative and positive) of each text token; or for complex application scenarios, each text token can be scored in multiple dimensions, such as three dimensions (emotional tendency, information importance, context relevance, etc.), and the text prediction result of whether each text token is retained is obtained according to the scores of multiple dimensions. The specific selection is based on actual needs, and one or more embodiments of this specification do not make any restrictions.

[0129] In practical applications, in most cases the text predictor will first generate text tokens, and then generate corresponding text masks based on these text tokens and specific task requirements for subsequent task processing.

[0130] The model training method provided in one or more embodiments of the present specification can train an initial text predictor based on sample text features of sample text data and text classification labels of the sample text features to obtain a text predictor, so that when a sparse first modal feature representation and a task text feature representation are decoded using a multimodal data processing model to obtain a text feature representation, the text predictor can be used to reduce the number of text feature representations and accelerate the training efficiency of the multimodal data processing model.

[0131] In specific implementation, in order to improve the overall performance of the text predictor, make the model more stable during the training process, and maintain good generalization ability, the text predictor can be divided into modules. The specific implementation method is as follows: The text predictor includes a linear network layer and a multi-layer perceptron network layer; The step of inputting the sample text features into an initial text predictor to obtain a predicted text classification label comprises: Inputting the sample text features into an initial text predictor; In the linear network layer, linear transformation is performed on the sample text features to obtain the sample text features after the linear transformation, and the sample text features after the linear transformation are input into the multi-layer perceptron network layer; In the multi-layer perceptron network layer, classification prediction is performed on the sample text features after the linear transformation to obtain a predicted text classification label of the sample text features.

[0132] Among them, the linear network layer (Linear Layer) is used to perform linear transformation on the sample text features, such as transforming the sample text features into 512-dimensional sample text features through matrix multiplication, that is, the sample text features after linear transformation.

[0133] The multilayer perceptron network layer contains one or more layers of nonlinear activation functions, which are used to perform nonlinear transformations on each sample text feature, so that the model can learn more complex feature representations, thereby achieving accurate classification predictions and obtaining predicted text classification labels for sample text features.

[0134] The model training method provided by one or more embodiments of the present specification, when the initial text predictor includes a linear network layer and a multi-layer perceptron network layer, inputs the sample text features into the linear network layer for linear transformation, and reduces the overfitting of the model through the fewer parameters of the linear network layer; then inputs the sample text features after the linear transformation into the multi-layer perceptron network layer, and performs nonlinear transformation through a nonlinear activation function, so that the model learns more complex feature representations and enhances the expression ability of the model; and through the overall implementation of the above two-layer structure, the overall performance of the model is improved.

[0135] In combination with the above embodiments, the network structure of the image predictor is compared with the network structure of the text predictor in that the image predictor introduces a visual processing network layer, while the text predictor does not use a visual processing network layer; this is because the image predictor processes visual data, which has the characteristics of high dimension, density, certain spatial structure and layout, long-distance dependence, etc. The visual processing network layer can effectively capture the interaction between any two positions in the image through the self-attention mechanism, rather than being limited to the local area, allowing the model to focus on different parts of the entire image, and can learn richer feature representations; while the text predictor processes text data, which has the characteristics of sequentiality, strong local dependence, and low dimension. The text predictor is generally based on the Transformer architecture, which can handle the sequentiality and long-distance dependence of the text very well, so there is no need to specifically introduce a visual processing network layer. Of course, in practical applications, if the visual processing network layer can solve the problems faced by the visual data through other implementations, the visual processing network layer can also be omitted.

[0136] See also Figure 6 , Figure 6 A schematic diagram of the structure of a text predictor in a model training method provided by an embodiment of this specification is shown.

[0137] Figure 6 The text predictor in includes a linear network layer (Linear Layer) and a multilayer perceptron network layer (Multilayer Perceptron).

[0138] Using the above example, the text token is input into the Linear Layer. In the Linear Layer, the input channel number is The feature dimension of the text token input to the Linear Layer is converted to 512 dimensions, that is, the input text token is mapped from the original dimension to a new 512-dimensional space, and the output dimension of the text token is 512 dimensions.

[0139] Then, the output of the Linear Layer is input into a 3-layer Multilayer Perceptron, and a nonlinear activation function is used to perform a nonlinear transformation on the 512-dimensional text token, converting it into a 2-dimensional output, and the prediction result is obtained based on the 2-dimensional output text token.

[0140] certainly, Figure 6 What is shown is only one implementation structure of the text predictor. In practical applications, different parts can be optimized or improved according to actual needs, such as changing the number of Vision Transformer Blocks, etc., without any limitation here.

[0141] Specifically, after obtaining the predicted sample data, the target loss value can be calculated according to the predicted sample data and the preset sample data, and the trained multimodal data processing model, the first modality sparse predictor, and the second modality sparse predictor can be updated according to the target loss value, that is, on the basis of the pre-trained multimodal large language model, the pre-trained first modality sparse predictor and the second modality sparse predictor are combined to perform end-to-end sparse instruction following fine-tuning training on the multimodal large language model, so as to improve the training efficiency of the multimodal large language model and the reasoning efficiency applied to the reasoning process.

[0142] In addition, during the model training process, the generated second modal feature unit set will also be sparsely processed. When calculating the target loss value, it is necessary to consider adding the sparse processing loss value of the second modal feature unit set to the target loss value calculation to achieve more effective feature selection and representation learning. The specific implementation method is as follows: The calculating the target loss value according to the predicted sample data and the preset sample data includes: Calculate a prediction loss value and a regularization loss value according to the first modal feature unit set, the first modal sparse feature unit set, the second modal feature unit set, the second modal sparse feature unit set, the prediction sample data and the preset sample data; A target loss value is calculated according to the predicted loss value and the regularized loss value.

[0143] Among them, the target loss value is composed of a standard cross entropy (CE) loss (i.e., a prediction loss value) and a regularization loss (i.e., a regularization loss value). The target loss value is used to update the multimodal large language model to enhance its generalization ability and stability.

[0144] Specifically, when dealing with joint tasks involving images and text, such as image-text matching or generation tasks, the model first extracts features from the input data through the encoder to obtain image tokens and text tokens, respectively. Next, the standard cross-entropy loss is used to measure the difference between the model prediction and the true label, which is a basic component in supervised learning and is used to guide the model to learn the correct classification or regression target. At the same time, a regularization loss based on mask and set size ratio is introduced, which takes into account the importance distribution of image tokens and output text tokens. By adjusting these mask and ratio parameters, the data contribution of different modalities can be effectively balanced to ensure that the model does not rely too much on the information of a specific modality. The entire training process combines these two losses and minimizes the total loss function through the optimization algorithm to update the model parameters.

[0145] The specific implementation is as follows: Calculating a prediction loss value and a regularization loss value according to the first modal feature unit set, the first modal sparse feature unit set, the second modal feature unit set, the second modal sparse feature unit set, the prediction sample data and the preset sample data, including: Calculating the predicted loss value according to the predicted sample data and the preset sample data; A regularization loss value is calculated according to the first modal feature unit set, the first modal sparse feature unit set, the second modal feature unit set, and the second modal sparse feature unit set.

[0146] Specifically, the predicted sample data refers to the output after being processed by a certain multimodal data processing model through the forward propagation stage in the training stage, and the preset sample data is the target sample data used for supervised learning during the training process of the multimodal data processing model, that is, the correct label or output.

[0147] After obtaining the predicted sample data, it is compared with the preset sample data and the prediction loss value is calculated, that is, the standard cross entropy loss function is used to measure the difference between the predicted sample data generated by the multimodal data processing model and the target sample data.

[0148] The regularization loss value is to balance the amount of information between different modalities and prevent the multimodal data processing model from overfitting to a specific modality. It is calculated based on the feature unit set of the first modality and the second modality, their sparse versions and the preset proportional relationship. For example, the first modality feature unit set is: all patches (image token set) of the original image, the first modality sparse feature unit set is: the important image token set retained after sparseness, the second modality feature unit set is: all tokens set of the text, the second modality sparse feature unit set is: the important text token set after sparseness, and the preset set ratio is: the ratio of the number of sparse feature units of each modality to the number of its original feature units. Then, according to the corresponding calculation formula (see Formula 5), ​​the regularization loss is calculated using the set size ratio defined above.

[0149] Among them, in the first round of iterative training of the model, the first modality sparse predictor can be understood as the image predictor obtained by training through the training steps of the above-mentioned image predictor; and the first modality sparse predictor after updating the training can be understood as the image predictor fine-tuned according to the target loss value; the second modality sparse predictor can be understood as the text predictor obtained by training through the training steps of the above-mentioned text predictor; and the second modality sparse predictor after updating the training can be understood as the text predictor fine-tuned according to the target loss value.

[0150] That is, when designing the loss function, a constrained regularization term can be added to ensure that the decisions made by updating the first modality sparse predictor obtained through training and the second modality sparse predictor can reduce the number of tokens by a preset ratio. This constrained regularization term penalizes too many non-zero eigenvalues, prompting the model to use as few features as possible to express information during the learning process. In addition, the weight of this sparse regularization term can be adjusted to balance the relationship between sparsity and prediction accuracy, ensuring that the final model can maintain efficient feature representation while achieving the expected prediction performance. Through this constrained regularization term, sparsification can be used as one of the training objectives, and the loss function can be modified accordingly, so that the model can better adapt to the needs of practical applications while improving its overall performance and efficiency.

[0151] Specifically, the constraint regularization term is expressed by the following formula 5: Formula 5 in, represents the sum operation, Indicates the number of tokens in the collection. represents the Frobenius norm, Indicates the length of the predefined output text token involved in the training update.

[0152] The model training method provided in one or more embodiments of this specification designs the training loss as a combination of standard cross entropy (CE) loss and regularization loss. When processing tasks involving different modalities such as images and texts, the standard CE loss ensures that the model can accurately learn basic classification or regression capabilities. It directly measures the difference between the model prediction and the true label, and guides the model to optimize its basic functions. However, relying solely on CE loss is often not enough to cope with complex data structures, especially in the presence of inconsistent information density or multimodal data; the introduction of regularization loss based on the first modal feature unit set, sparse feature unit set and preset ratio greatly enhances the generalization ability and stability of the model. By dynamically adjusting the importance distribution of image tokens and text tokens, the contribution between different modalities is effectively balanced to avoid a certain modality dominating the learning process of the model. For example, in the image-text matching task, the regularization loss helps to accurately adjust the information weight between the image and the text, making the model more accurate in understanding and generating descriptions. In addition, this dual loss mechanism can also alleviate the overfitting problem, especially when the training data is limited, to prevent the model from overfitting the details in the training set, thereby improving the adaptability to unseen data.

[0153] In one or more embodiments of this specification, after obtaining the first modal feature representation or decoding the second modal feature representation, in order to improve the training efficiency and subsequent reasoning efficiency of the multimodal data processing model, the first modal feature representation will be sparsely processed using the first modal sparse predictor, or the decoded second modal feature representation will be sparsely processed using the second modal sparse predictor. In order to ensure that the sparsely processed first modal feature representation or the sparsely processed decoded second modal feature representation is not considered when calculating the probability distribution, and to ensure that the learning and prediction of the model are more in line with actual needs, the probability sampling function (GumbelSoftmax function) and the probability adjustment function (MaskedSoftmax function) can participate in the model training process. The specific implementation method is as follows: The method of performing sparse processing using the first modal sparse predictor to obtain a first modal sparse feature unit set includes: In the forward propagation stage, a first modal sparse predictor is used to generate a first decision matrix, and the first decision matrix is ​​subjected to approximate discrete sampling processing to obtain a first approximate decision matrix; Performing sampling using a decision function on a preset dimension of the first approximate decision matrix to generate a first mask matrix; Filtering the first modal feature unit set according to the first mask matrix to obtain a first modal sparse feature unit set; In the back propagation phase of the training phase, a target loss value is calculated according to the predicted sample data and the preset sample data, and the multimodal data processing model and the first modality sparse predictor are updated and trained according to the target loss value, including: In the back-propagation stage, gradient calculation is performed based on the first approximate decision matrix.

[0154] Among them, the first decision matrix is ​​subjected to approximate discrete sampling processing using a discrete distribution normalized exponential function, and the discrete distribution normalized exponential function can be understood as a GumbelSoftmax function. In practical applications, the discrete distribution normalized exponential function can also be replaced by other functions with the same function as the GumbelSoftmax function, and this specification does not impose any restrictions on this. When the decision function is used to sample on the preset dimension of the first approximate decision matrix to generate the first mask matrix, the second probability processing function can be used to adjust the probability distribution to obtain a more accurate first mask matrix. Among them, the second probability processing function is a MaskedSoftmax function.

[0155] Specifically, in the forward propagation stage, when the first modal sparse predictor is used to perform sparse processing to obtain the first modal sparse feature unit set, the first modal feature unit set to be calculated is input into the first modal sparse predictor for sparse processing to obtain the first modal feature unit classification result of the first modal feature unit, then when the first modal feature unit classification result is the selection probability distribution of each first modal feature unit, the GumbelSoftmax function is used to perform soft sampling from the probability distribution, and by introducing continuity, the gradient can flow through while retaining the essence of discrete selection. Based on the soft sampling result, the first modal feature unit finally selected is determined to form a preliminary decision list, and a mask matrix is ​​constructed according to the decision list; wherein, for each first modal feature unit decided to be discarded, the corresponding element in the mask matrix can be set to 0, and for each first modal feature unit decided to be retained, the corresponding element in the mask matrix can be set to 1.

[0156] Then, the MaskedSoftmax function is used to check the mask matrix. For the elements marked as 0 in the mask matrix, the corresponding attention weights will be set to the minimum value, and for the elements marked as 1 in the mask matrix, the corresponding attention weights will be set to a larger value. The adjusted attention weights are then used to generate the final first mask matrix. Finally, the first modal feature unit set is screened according to the first mask matrix to obtain the first modal sparse feature unit set.

[0157] Similarly, the specific implementation of using the second modality sparse predictor to perform sparse processing to obtain the second modality sparse feature unit set can refer to the specific implementation steps of using the second modality sparse predictor to perform sparse processing to obtain the second modality sparse feature unit set. That is, the processing method of obtaining the second modality sparse feature unit set in the forward propagation stage of model training using the GumbelSoftmax function and the MaskedSoftmax function is the same as the processing method of obtaining the first modality sparse feature unit set, which will not be repeated here.

[0158] The definition of the MaskedSoftmax function is shown in the following formula 6: Formula 6 in, Represents the input of the original Softmax function, ,at the same time , Indicates participation in the multimodal large language model The number of tokens in the text token set calculated by the layer; in the calculation of the MaskedSoftmax function, keep ,Right now The values ​​on the diagonal of the matrix are kept as 1.

[0159] The definition of the GumbelSoftmax function in the forward propagation stage is shown in Formula 7 below, and the definition in the backward propagation stage is shown in Formula 8 below: Formula 7 Formula 8 in, is the temperature of the GumbelSoftmax function, is the intermediate output, is the objective function of training (target loss value).

[0160] The model training method provided in one or more embodiments of this specification, in the training of the multimodal data processing model combining the first modality sparse predictor and the second modality sparse predictor, uses the GumbelSoftmax function to generate decisions and process to obtain a mask matrix, and uses the MaskedSoftmax function to replace the Softmax function to isolate the influence of unimportant tokens (text tokens or image tokens). Through the operation of these two functions, the model can not only make more flexible and differentiable discrete choices during the training process, but also ensure that these choices will not interfere with the truly important tokens. It helps to improve the learning efficiency of the model, enhance the generalization ability, and ensure that the final generated output is more accurate and meaningful.

[0161] Since the forward process of model training uses the GumbelSoftmax function to maintain the continuity of the gradient flow, thereby allowing end-to-end gradient direction propagation, this method is also needed during the reverse propagation process of the model to ensure that the gradient can be smoothly propagated back. The specific implementation method is as follows: In the back propagation stage, gradient calculation is performed based on the first approximate decision matrix, including: In the back propagation stage, a gradient pass-through estimator is used to perform gradient calculation on the first decision matrix to obtain approximate gradient information; Calculating, according to the approximate gradient information and the target loss value, the gradients of the parameters in the multimodal data processing model, the first modality sparse predictor, and the second modality sparse predictor relative to the target loss value; According to the gradients of the parameters in the multimodal data processing model, the first modal sparse predictor, and the second modal sparse predictor relative to the target loss value, the parameters of the multimodal data processing model, the first modal sparse predictor, and the second modal sparse predictor are updated until a preset stopping condition is reached.

[0162] Among them, the gradient straight-through estimator can be understood as STE, straight-through estimator.

[0163] Specifically, when the target loss value is calculated, the gradient of each parameter in the model is calculated based on the target loss value, and the gradient information provided by STE and GumbelSoftmax function is used to pass the gradient back to the multimodal data processing model, the first modality sparse predictor, and the second modality sparse predictor to achieve update training of the multimodal data processing model, the first modality sparse predictor, and the second modality sparse predictor.

[0164] The model training method provided by one or more embodiments of this specification is a back propagation process for updating the training of the multimodal data processing model, the first modality sparse predictor, and the second modality sparse predictor. The target loss value is used to guide the gradient calculation to obtain the gradient of each parameter of the model, and then the gradient information provided by STE and GumbelSoftmax is used to back propagate the gradient, which solves the gradient problem in discrete selection tasks (text token selection tasks or image token selection tasks) and ensures that the model can effectively learn meaningful feature representations throughout the training process, thereby improving its generalization ability and final performance. In this way, even if discrete selection is involved, the model can be trained in an end-to-end manner to better capture complex patterns in the data.

[0165] Specifically, the traditional Softmax function assumes that all inputs are valid, that is, the probability of each category can be calculated. However, in some cases, it may be necessary to ignore or mask part of the input (for example, in some tasks, some output categories are invalid). In this case, MaskedSoftmax can be used to set a mask to tell the function which positions should be ignored, so that only the valid parts are normalized. This helps to improve the accuracy and efficiency of the model without wasting computing resources on invalid outputs.

[0166] Moreover, the traditional argmax operation is a non-continuous operation that selects a single output with the highest probability, and this selection process does not provide useful gradient information to the back-propagation algorithm. This means that when training the model, if the final choice is made by argmax, then the parameters cannot be effectively updated to optimize the choice. The GumbelSoftmax function provides a way to approximate the sampling of discrete distributions while maintaining the continuity of the gradient flow. By introducing the temperature parameter, it can control the sharpness of the output distribution, so that a smooth probability distribution can be obtained during training, and a more deterministic choice can be obtained by lowering the temperature parameter during the inference phase. In addition, the gradient pass-through estimator allows the gradient to be passed from the output layer to the input layer, even when discrete choices are involved. This allows the entire network to be trained on an end-to-end basis and to more effectively learn and optimize tasks involving discrete choices.

[0167] Therefore, in the model training method provided in one or more embodiments of this specification, the GumbelSoftmax function and the MaskedSoftmax function will be applied to the forward propagation and backward propagation stages of the model to complete the model training.

[0168] Optionally, at least some of the multiple model layers are used to output probability distribution using a first probability processing function in the inference stage, and to output probability distribution after mask processing using a second probability processing function in the forward propagation stage.

[0169] The first probability processing function is a Softmax function, and the second probability processing function is a MaskedSoftmax function.

[0170] Specifically, in the inference phase, usually all possible outputs are valid, so the Softmax function can be used directly to calculate the probability of each category. Since the inference process focuses more on efficiency, Softmax is easier to calculate than Masked Softmax, and does not require additional mask operations, which helps to speed up the inference. In the training phase, in some application scenarios, such as sequence generation and attention mechanisms, not all outputs are valid or relevant. For example, when predicting the next word in an autoregressive model, the already generated part should no longer be used as a candidate. At this time, using Masked Softmax can help the model ignore these invalid options; especially in tasks involving future information, such as text generation, if it is not restricted, it may cause the model to "see" future data points, causing information leakage. Masked Softmax can effectively prevent this from happening, ensuring that the model can only make decisions based on existing information.

[0171] Corresponding to the above method embodiment, this specification also provides a multimodal data processing device embodiment, Figure 7 FIG. 1 shows a schematic diagram of the structure of a multimodal data processing device provided by an embodiment of the present specification. Figure 7 As shown, the device comprises: The data processing module 702 is configured to input the data to be processed into the multimodal data processing model so that the multimodal data processing model outputs target data after processing in the inference phase, wherein the data to be processed includes at least first modality data, the target data includes at least second modality data, and the multimodal data processing model includes a plurality of model layers, each model layer performs corresponding calculations in a corresponding inference phase based on a set of feature units; A first sparse module 704 is configured to, in the inference stage, for a target model layer among the multiple model layers, perform sparse processing on the first modal feature unit set to be calculated by using a first modal sparse predictor to obtain a first modal sparse feature unit set, wherein the number of feature units in the first modal sparse feature unit set is less than the number of feature units in the first modal feature unit set; The data calculation module 706 is configured to input the first modal sparse feature unit set into the target model layer so that the target model layer performs calculation based on the first modal sparse feature unit set.

[0172] Optionally, the device further includes: a second sparse module, configured to: For the target model layer, the second modality feature unit set to be calculated is sparsely processed by using a second modality sparse predictor to obtain a second modality sparse feature unit set; inputting the second modality sparse feature unit set into the target model layer so that the target model layer performs calculation based on the second modality sparse feature unit set, The number of feature units in the second modality sparse feature unit set is less than the number of feature units in the second modality feature unit set.

[0173] Optionally, the reasoning stage includes an input processing stage; The first sparse module 704 is further configured to: The first modal sparse feature unit set is input into the target model layer so that the target model layer performs a first calculation based on the first modal sparse feature unit set during the input processing stage, wherein the first modal feature unit set is obtained based on the first modal data of the data to be processed.

[0174] Optionally, the reasoning stage includes an output processing stage; The second sparse module is further configured as follows: Inputting the second modal sparse feature unit set into the target model layer so that the target model layer performs a second calculation based on the second modal sparse feature unit set in the output processing stage, wherein the second modal feature unit set is obtained based on the second modal data of the target data; and / or The data calculation module is further configured as follows: The first modal sparse feature unit set is input into the target model layer so that the target model layer performs a second calculation based on the first modal sparse feature unit set in the output processing stage.

[0175] Optionally, the first modality includes an image and the second modality includes text.

[0176] Optionally, the first sparse module 704 is further configured to: generating a first decision matrix using a first modal sparse predictor; Performing sampling using a decision function on a preset dimension of the first decision matrix to generate a first mask matrix; Filtering the first modal feature unit set according to the first mask matrix to obtain a first modal sparse feature unit set; and / or The second sparse module is further configured as follows: generating a second decision matrix using a second modality sparse predictor; Performing sampling using a decision function on a preset dimension of the second decision matrix to generate a second mask matrix; The second modality feature unit set is screened according to the second mask matrix to obtain a second modality sparse feature unit set.

[0177] Optionally, the first modality sparse predictor includes a linear network layer, a visual processing network layer, and a multi-layer perceptron network layer, and the second modality sparse predictor includes a linear network layer and a multi-layer perceptron network layer.

[0178] The above is a schematic scheme of a multimodal data processing device of this embodiment. It should be noted that the technical scheme of the multimodal data processing device and the technical scheme of the multimodal data processing method described above belong to the same concept, and the details not described in detail in the technical scheme of the multimodal data processing device can be found in the description of the technical scheme of the multimodal data processing method described above.

[0179] Corresponding to the above method embodiment, this specification also provides a model training device embodiment, Figure 8 FIG. 1 shows a schematic diagram of a multi-model training device provided by an embodiment of the present specification. Figure 8 As shown, the device comprises: The data acquisition module 802 is configured to acquire a training sample data set, wherein the training sample data set includes a plurality of sample data to be processed, and the sample data to be processed includes at least first modality data; The data processing module 804 is configured to input the sample data to be processed into the multimodal data processing model, so that the multimodal data processing model outputs predicted sample data after processing in the forward propagation phase in the training phase, wherein the predicted sample data at least includes the second modality data, and the multimodal data processing model includes a plurality of model layers, each model layer performs corresponding calculations in a corresponding training phase based on a feature unit set, The first sparse module 806 is configured to perform sparse processing on the first modal feature unit set to be calculated using the first modal sparse predictor for the target model layer in the multiple model layers in the forward propagation stage of the training stage to obtain a first modal sparse feature unit set, wherein the number of feature units in the first modal sparse feature unit set is less than the number of feature units in the first modal feature unit set. The data calculation module 808 is configured to input the first modal sparse feature unit set into the target model layer so that the target model is calculated based on the first modal sparse feature unit set. The training module 810 is configured to calculate the target loss value according to the predicted sample data and the preset sample data in the back propagation stage of the training stage, and update the training of the multimodal data processing model and the first modality sparse predictor according to the target loss value.

[0180] Optionally, the device further includes a second sparse module configured to: In the back propagation phase of the training phase, for the target model layer, the second modality feature unit set to be calculated is sparsely processed by using the second modality sparse predictor to obtain the second modality sparse feature unit set, Inputting the second modality sparse feature unit set into the target model layer so that the target model layer performs calculation based on the second modality sparse feature unit set, wherein the number of feature units in the second modality sparse feature unit set is less than the number of feature units in the second modality feature unit set; The training module 810 is further configured to: In the back propagation stage of the training stage, a target loss value is calculated based on the predicted sample data and the preset sample data, and the multimodal data processing model, the first modality sparse predictor, and the second modality sparse predictor are updated and trained based on the target loss value.

[0181] Optionally, the first sparse module 806 is further configured to: Generate a first decision matrix using a first modal sparse predictor, and perform approximate discrete sampling processing on the first decision matrix to obtain a first approximate decision matrix; Performing sampling using a decision function on a preset dimension of the first approximate decision matrix to generate a first mask matrix; Filtering the first modal feature unit set according to the first mask matrix to obtain a first modal sparse feature unit set; The training module 810 is further configured to: In the back propagation stage, gradient calculation is performed based on the first approximate decision matrix.

[0182] Optionally, at least some of the multiple model layers are used to output probability distribution using a first probability processing function in the inference stage, and to output probability distribution after mask processing using a second probability processing function in the forward propagation stage.

[0183] Optionally, the training module 810 is further configured to: Calculate a prediction loss value and a regularization loss value according to the first modal feature unit set, the first modal sparse feature unit set, the second modal feature unit set, the second modal sparse feature unit set, the prediction sample data and the preset sample data; A target loss value is calculated according to the predicted loss value and the regularized loss value.

[0184] Optionally, the training module 810 is further configured to: Calculating the predicted loss value according to the predicted sample data and the preset sample data; The regularization loss value is calculated according to the first modal feature unit set, the first modal sparse feature unit set, the second modal feature unit set, and the second modal sparse feature unit set.

[0185] The above is a schematic scheme of a model training device of this embodiment. It should be noted that the technical scheme of the model training device and the technical scheme of the above-mentioned model training method belong to the same concept, and the details not described in detail in the technical scheme of the model training device can be referred to the description of the technical scheme of the above-mentioned model training method.

[0186] Fig. 9 The structure block diagram of a computing device 900 provided according to an embodiment of the present specification is shown. The components of the computing device 900 include but are not limited to a memory 910 and a processor 920. The processor 920 is connected to the memory 910 via a bus 930, and a database 950 is used to store data.

[0187] The computing device 900 also includes an access device 940 that enables the computing device 900 to communicate via one or more networks 960. Examples of these networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 940 may include one or more of any type of network interface (e.g., a network interface card (NIC)) of wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a world-wide interoperability for microwave access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.

[0188] In one embodiment of the present specification, the above components of the computing device 900 and Fig. 9 Other components not shown in the figure may also be connected to each other, for example, via a bus. It should be understood that Fig. 9 The computing device structure block diagram shown is only for the purpose of illustration, and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0189] The computing device 900 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, a personal digital assistant, a laptop computer, a notebook computer, a netbook, etc.), a mobile phone (e.g., a smart phone), a wearable computing device (e.g., a smart watch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or a personal computer (PC). The computing device 900 may also be a mobile or stationary server.

[0190] Among them, the processor 920 is used to execute the following computer program / instructions, which, when executed by the processor, implement the steps of the above-mentioned multimodal data processing method or model training method.

[0191] The above is a schematic scheme of a computing device of this embodiment. It should be noted that the technical scheme of the computing device and the technical scheme of the multimodal data processing method or model training method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the computing device can be referred to the description of the technical scheme of the multimodal data processing method or model training method mentioned above.

[0192] An embodiment of the present specification also provides a computer-readable storage medium storing a computer program / instruction, which implements the steps of the above-mentioned multimodal data processing method or model training method when executed by a processor.

[0193] The above is a schematic scheme of a computer-readable storage medium of this embodiment. It should be noted that the technical scheme of the storage medium and the technical scheme of the multimodal data processing method or model training method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the storage medium can be referred to the description of the technical scheme of the multimodal data processing method or model training method mentioned above.

[0194] An embodiment of the present specification also provides a computer program product, including a computer program / instruction, which implements the steps of the above-mentioned multimodal data processing method or model training method when executed by a processor.

[0195] The above is a schematic scheme of a computer program product of this embodiment. It should be noted that the technical scheme of the computer program product and the technical scheme of the multimodal data processing method or model training method mentioned above belong to the same concept, and the details not described in detail in the technical scheme of the computer program product can be found in the description of the technical scheme of the multimodal data processing method or model training method mentioned above.

[0196] The above is a description of a specific embodiment of the specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require the specific order or continuous order shown to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0197] The computer program / instruction includes computer program code, which may be in source code form, object code form, executable file or some intermediate form, etc. The computer readable medium may include: any entity or device capable of carrying the computer program code, recording medium, USB flash drive, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electric carrier signal, telecommunication signal and software distribution medium, etc. It should be noted that the content contained in the computer readable medium may be appropriately increased or decreased according to the requirements of patent practice. For example, in some regions, according to patent practice, computer readable media do not include electric carrier signals and telecommunication signals.

[0198] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that the embodiments of this specification are not limited by the order of the actions described, because according to the embodiments of this specification, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the embodiments of this specification.

[0199] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0200] The preferred embodiments of this specification disclosed above are only used to help explain this specification. The optional embodiments do not describe all the details in detail, nor do they limit the invention to only the specific implementation methods described. Obviously, many modifications and changes can be made according to the content of the embodiments of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the embodiments of this specification, so that technicians in the relevant technical field can well understand and use this specification. This specification is only limited by the claims and their full scope and equivalents.

Claims

1. A multimodal data processing method, characterized in that: include: Inputting the data to be processed into the multimodal data processing model so that the multimodal data processing model outputs target data after processing in the inference phase, wherein the data to be processed includes at least first modality data, the target data includes at least second modality data, and the multimodal data processing model includes a plurality of model layers, each model layer performing corresponding calculations in a corresponding inference phase based on a set of feature units; In the inference stage, for a target model layer among the multiple model layers, a first modal feature unit set to be calculated is sparsely processed by using a first modal sparse predictor to obtain a first modal sparse feature unit set, wherein the number of feature units in the first modal sparse feature unit set is less than the number of feature units in the first modal feature unit set; The first modal sparse feature unit set is input into the target model layer so that the target model layer performs calculation based on the first modal sparse feature unit set.

2. The multimodal data processing method according to claim 1, characterized in that: Also includes: For the target model layer, the second modality feature unit set to be calculated is sparsely processed by using a second modality sparse predictor to obtain a second modality sparse feature unit set; inputting the second modality sparse feature unit set into the target model layer so that the target model layer performs calculation based on the second modality sparse feature unit set, The number of feature units in the second modality sparse feature unit set is less than the number of feature units in the second modality feature unit set.

3. The multimodal data processing method according to claim 1, characterized in that: The reasoning stage includes an input processing stage; The step of inputting the first modal sparse feature unit set into the target model layer so that the target model layer performs calculation based on the first modal sparse feature unit set includes: The first modal sparse feature unit set is input into the target model layer so that the target model layer performs a first calculation based on the first modal sparse feature unit set during the input processing stage, wherein the first modal feature unit set is obtained based on the first modal data of the data to be processed.

4. The multimodal data processing method according to claim 2, characterized in that: The reasoning stage includes an output processing stage; The step of inputting the second modal sparse feature unit set into the target model layer so that the target model layer performs calculation based on the second modal sparse feature unit set includes: Inputting the second modal sparse feature unit set into the target model layer so that the target model layer performs a second calculation based on the second modal sparse feature unit set in the output processing stage, wherein the second modal feature unit set is obtained based on the second modal data of the target data; and / or The step of inputting the first modal sparse feature unit set into the target model layer so that the target model layer performs calculation based on the first modal sparse feature unit set includes: The first modal sparse feature unit set is input into the target model layer so that the target model layer performs a second calculation based on the first modal sparse feature unit set in the output processing stage.

5. The multimodal data processing method according to claim 1, characterized in that: The first modality includes an image and the second modality includes text.

6. The multimodal data processing method according to claim 2, characterized in that: The method of performing sparse processing using the first modal sparse predictor to obtain a first modal sparse feature unit set includes: generating a first decision matrix using a first modal sparse predictor; Performing sampling using a decision function on a preset dimension of the first decision matrix to generate a first mask matrix; Filtering the first modal feature unit set according to the first mask matrix to obtain a first modal sparse feature unit set; and / or The sparse processing is performed using the second modality sparse predictor to obtain a second modality sparse feature unit set, including: generating a second decision matrix using a second modality sparse predictor; Performing sampling using a decision function on a preset dimension of the second decision matrix to generate a second mask matrix; The second modality feature unit set is screened according to the second mask matrix to obtain a second modality sparse feature unit set.

7. The multimodal data processing method according to claim 2, characterized in that: The first modality sparse predictor includes a linear network layer, a visual processing network layer, and a multi-layer perceptron network layer, and the second modality sparse predictor includes a linear network layer and a multi-layer perceptron network layer.

8. A model training method, characterized in that: include: Acquire a training sample data set, wherein the training sample data set includes a plurality of sample data to be processed, and the sample data to be processed includes at least first modality data; The sample data to be processed is input into a multimodal data processing model, so that the multimodal data processing model outputs predicted sample data after being processed in a forward propagation phase in a training phase, wherein the predicted sample data at least includes second modality data, and the multimodal data processing model includes a plurality of model layers, each model layer performs corresponding calculations in a corresponding training phase based on a set of feature units, In the forward propagation phase of the training phase, for a target model layer among the multiple model layers, a first modal feature unit set to be calculated is sparsely processed by using a first modal sparse predictor to obtain a first modal sparse feature unit set, wherein the number of feature units in the first modal sparse feature unit set is less than the number of feature units in the first modal feature unit set; Inputting the first modal sparse feature unit set into the target model layer so that the target model is calculated based on the first modal sparse feature unit set; In the back propagation stage of the training stage, a target loss value is calculated based on the predicted sample data and the preset sample data, and the multimodal data processing model and the first modality sparse predictor are updated and trained based on the target loss value.

9. The model training method according to claim 8, characterized in that: Also includes: In the forward propagation phase of the training phase, for the target model layer, the second modality feature unit set to be calculated is sparsely processed by using the second modality sparse predictor to obtain the second modality sparse feature unit set, Inputting the second modality sparse feature unit set into the target model layer so that the target model layer performs calculation based on the second modality sparse feature unit set, wherein the number of feature units in the second modality sparse feature unit set is less than the number of feature units in the second modality feature unit set; The back propagation phase in the training phase calculates a target loss value according to the predicted sample data and the preset sample data, and updates and trains the multimodal data processing model and the first modality sparse predictor according to the target loss value, including: In the back propagation stage of the training stage, a target loss value is calculated based on the predicted sample data and the preset sample data, and the multimodal data processing model, the first modality sparse predictor, and the second modality sparse predictor are updated and trained based on the target loss value.

10. The model training method according to claim 8 or 9, characterized in that: The method of performing sparse processing using the first modal sparse predictor to obtain a first modal sparse feature unit set includes: In the forward propagation stage, a first decision matrix is ​​generated by using a first modal sparse predictor, and the first decision matrix is ​​subjected to approximate discrete sampling processing to obtain a first approximate decision matrix; Performing sampling using a decision function on a preset dimension of the first approximate decision matrix to generate a first mask matrix; Filtering the first modal feature unit set according to the first mask matrix to obtain a first modal sparse feature unit set; In the back propagation phase of the training phase, a target loss value is calculated according to the predicted sample data and the preset sample data, and the multimodal data processing model and the first modality sparse predictor are updated and trained according to the target loss value, including: In the back propagation stage, gradient calculation is performed based on the first approximate decision matrix.

11. The model training method according to claim 10, characterized in that: At least some of the multiple model layers are used to output probability distribution using a first probability processing function in the inference stage, and to output probability distribution after mask processing using a second probability processing function in the forward propagation stage.

12. The model training method according to claim 9, characterized in that: The calculating the target loss value according to the predicted sample data and the preset sample data includes: Calculate a prediction loss value and a regularization loss value according to the first modal feature unit set, the first modal sparse feature unit set, the second modal feature unit set, the second modal sparse feature unit set, the prediction sample data and the preset sample data; A target loss value is calculated according to the predicted loss value and the regularized loss value.

13. The model training method according to claim 12, characterized in that: The step of calculating the prediction loss value and the regularization loss value according to the first modal feature unit set, the first modal sparse feature unit set, the second modal feature unit set, the second modal sparse feature unit set, the prediction sample data and the preset sample data includes: Calculating the predicted loss value according to the predicted sample data and the preset sample data; The regularization loss value is calculated according to the first modal feature unit set, the first modal sparse feature unit set, the second modal feature unit set, and the second modal sparse feature unit set.

14. A multimodal data processing device, characterized in that: include: A data processing module is configured to input the data to be processed into the multimodal data processing model, so that the multimodal data processing model outputs target data after processing in the inference phase, wherein the data to be processed includes at least first modality data, the target data includes at least second modality data, and the multimodal data processing model includes a plurality of model layers, each model layer performs corresponding calculations in a corresponding inference phase based on a set of feature units; A first sparse module is configured to, in the inference stage, for a target model layer among the multiple model layers, perform sparse processing on the first modal feature unit set to be calculated by using a first modal sparse predictor to obtain a first modal sparse feature unit set, wherein the number of feature units in the first modal sparse feature unit set is less than the number of feature units in the first modal feature unit set; The data calculation module is configured to input the first modal sparse feature unit set into the target model layer so that the target model layer performs calculation based on the first modal sparse feature unit set.

15. A model training device, characterized in that: include: A data acquisition module is configured to acquire a training sample data set, wherein the training sample data set includes a plurality of sample data to be processed, and the sample data to be processed includes at least first modality data; The data processing module is configured to input the sample data to be processed into the multimodal data processing model, so that the multimodal data processing model outputs predicted sample data after processing in the forward propagation phase in the training phase, wherein the predicted sample data at least includes the second modality data, and the multimodal data processing model includes a plurality of model layers, each model layer performs corresponding calculations in a corresponding training phase based on a set of feature units, A first sparse module is configured to perform sparse processing on a first modal feature unit set to be calculated using a first modal sparse predictor for a target model layer in the multiple model layers during a forward propagation phase in the training phase, to obtain a first modal sparse feature unit set, wherein the number of feature units in the first modal sparse feature unit set is less than the number of feature units in the first modal feature unit set, a data calculation module, configured to input the first modal sparse feature unit set into the target model layer so that the target model performs calculation based on the first modal sparse feature unit set, The training module is configured to calculate a target loss value according to the predicted sample data and the preset sample data in the back propagation stage of the training stage, and update the training of the multimodal data processing model and the first modality sparse predictor according to the target loss value.

16. A computing device, characterized in that: include: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions. When the computer program / instructions are executed by the processor, the steps of the method described in any one of claims 1 to 13 are implemented.

17. A computer-readable storage medium, characterized in that: It stores a computer program / instruction, which implements the steps of the method described in any one of claims 1 to 13 when executed by a processor.

18. A computer program product, characterized in that The method comprises a computer program / instruction which, when executed by a processor, implements the steps of the method according to any one of claims 1 to 13.

Images