Inference method, device, equipment, medium and program product of multi-modal data

Abstract

This application discloses a method, apparatus, device, medium, and program product for inference of multimodal data, relating to the field of artificial intelligence technology. The method includes: acquiring a first token sequence of multimodal data; obtaining indicator parameters of non-text tokens in the first token sequence based on the tail token, whereby the indicator parameters indicate the importance of non-text tokens to the accuracy of the inference result of the multimodal data; compressing the non-text tokens in the first token sequence according to the indicator parameters to obtain a second token sequence; and performing inference based on the second token sequence to obtain the inference result of the multimodal data. This method reduces the length of the second token sequence by compressing the number of non-text tokens, thereby reducing the complexity of inference based on the second token sequence and accelerating inference of multimodal data.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to reasoning methods, apparatus, devices, media, and program products for multimodal data. Background Technology

[0002] In the field of artificial intelligence (AI) technology, scenarios involving reasoning based on multimodal data are becoming increasingly common. Multimodality includes textual and non-textual modalities. Non-textual modalities include, but are not limited to, audio, image, and video modalities; image and video modalities can be collectively referred to as visual modalities. In the process of reasoning with multimodal data, the data can be transformed into a sequence of tokens. Reasoning is then performed on these token sequences to obtain the reasoning result. For example, textual modal data can be converted into text tokens through word segmentation, and image modal data can be converted into image tokens through a visual encoder.

[0003] However, as the proportion of non-textual modal data in multimodal data continues to increase, the number of non-textual tokens derived from non-textual modal data also increases, leading to a continuous increase in the length of token sequences. The longer the token sequence, the higher the computational complexity during inference, resulting in slower inference speeds for multimodal data. Therefore, accelerating the inference process for multimodal data is an urgent problem to be solved. Summary of the Invention

[0004] This application provides a method, apparatus, device, medium, and program product for reasoning multimodal data, used to accelerate the reasoning process of multimodal data.

[0005] Firstly, a method for inference using multimodal data is provided. This method includes: obtaining a first token sequence of multimodal data, the first token sequence including non-text tokens; obtaining indicator parameters of the non-text tokens based on the tail token, the tail token being the last element of the first token sequence and representing a feature of the multimodal data, the indicator parameters indicating the importance of the non-text tokens to the accuracy of the inference result of the multimodal data; compressing the non-text tokens in the first token sequence according to the indicator parameters to obtain a second token sequence; and performing inference based on the second token sequence to obtain the inference result of the multimodal data.

[0006] This method reduces the length of the second token sequence by compressing the number of non-text tokens. A shorter second token sequence results in lower complexity for inference based on it, thus accelerating inference for multimodal data. The indicator parameters representing the importance of non-text tokens are obtained from the tail token. Since the tail token represents the characteristics of multimodal data, it incorporates the overall information of the multimodal data. Therefore, the indicator parameters obtained from the tail token take into account the impact of multimodal data on non-text tokens, improving the accuracy of the indicator parameters for non-text tokens, and consequently improving the accuracy of compressing the number of non-text tokens according to the indicator parameters.

[0007] Furthermore, compared to methods that determine the importance of non-text tokens based on the first token sequence, this method determines the importance of non-text tokens based on the tail token. The tail token has a smaller data volume than the multimodal data, reducing the computational cost of determining the importance of non-text tokens and further accelerating the inference process for multimodal data. The importance of a non-text token is also its significance to the accuracy of the inference result for the multimodal data. Compared to methods that determine the importance of non-text tokens based on attention scores provided by attention operators, this method does not rely on attention scores, making it adaptable to attention operators that cannot provide attention scores.

[0008] In one possible implementation, the method of obtaining the indicator parameters of the non-text token based on the tail token includes: obtaining a query (Q) matrix generated based on the tail token and a key (K) matrix generated based on the non-text token, where the Q matrix represents the information of interest in the reasoning process and the K matrix represents the features of the corresponding token; and obtaining the indicator parameters of the non-text token based on the similarity between the Q matrix and the K matrix.

[0009] Therefore, the similarity between non-text tokens and tail tokens can be accurately and quickly calculated using the Q and K matrices. The greater the similarity with the tail token, the higher the importance of the non-text token, thus enabling the rapid acquisition of non-text token metrics based on similarity. Optionally, the Q and K matrices are used to calculate attention scores in the attention mechanism. When this method is based on attention mechanism reasoning, the Q and K matrices output by the attention operator can be directly used without separate calculation, thereby accelerating the acquisition of metrics.

[0010] In one possible implementation, the indicator parameters for non-text tokens are obtained based on the similarity between the Q matrix and the K matrix. This includes: using the similarity between the Q matrix and the K matrix as the indicator parameter for non-text tokens; or, performing pooling or smoothing processing on the similarity between the Q matrix and the K matrix, and using the processed similarity between the Q matrix and the K matrix as the indicator parameter for any non-text token. Therefore, similarity can be used directly as the indicator parameter, or a variant of similarity can be used, improving the flexibility in obtaining the indicator parameter. The variant of similarity refers to the similarity after pooling or smoothing processing.

[0011] In one possible implementation, the Q matrix is ​​a compressed version of the K matrix. The uncompressed Q matrix has a greater number of attention heads than the K matrix, while the compressed Q matrix has the same number of attention heads as the K matrix. This ensures that the Q matrix and K matrix have the same number of attention heads, facilitating the calculation of their similarity and improving the efficiency of similarity calculation.

[0012] In one possible implementation, obtaining the first token sequence of multimodal data includes: inputting the multimodal data into a large multimodal model, which includes N Transformer layers, where N is a positive integer; obtaining the first token sequence based on the output of the (m-1)th Transformer layer, where m is a positive integer less than or equal to N; and performing inference based on the second token sequence to obtain the inference result of the multimodal data includes: inputting the second token sequence into the m-th Transformer layer of the large multimodal model, and obtaining the inference result of the multimodal data based on the output of the large multimodal model. This allows the method to combine a large multimodal model to perform inference on multimodal data, thereby shortening the inference time of the large multimodal model.

[0013] In one possible implementation, the model variables in the Transformer at layer (m-1) and before layer (m-1) are matched with the size of the first token sequence, and the model variables in the Transformer at layer m and after layer m are matched with the size of the second token sequence. Thus, in scenarios involving large multimodal models, the model variables change with the length of the token sequence, avoiding problems caused by incorrect model variable sizes, and enabling this method to flexibly adapt to large multimodal models.

[0014] In one possible implementation, compressing the non-text tokens in the first token sequence according to index parameters to obtain the second token sequence includes: compressing the non-text tokens in the first token sequence according to index parameters and a compression ratio, where the compression ratio is the ratio of the total number of non-text tokens in the second token sequence to the total number of non-text tokens in the first token sequence. Thus, more accurate compression of non-text tokens is achieved by applying the compression ratio.

[0015] In one possible implementation, the values ​​of m and compression ratio are the values ​​of the multimodal large model under the condition that the inference accuracy of the multimodal large model is greater than an accuracy threshold, and the inference time of the multimodal large model is less than a time threshold. Therefore, performing inference according to the values ​​of m and compression ratio can guarantee that the inference accuracy is greater than the accuracy threshold and the inference time is less than the time threshold.

[0016] In one possible implementation, the method further includes: identifying non-text tokens in the first token sequence based on a mask sequence corresponding to the first token sequence, wherein the length of the mask sequence is the same as the length of the first token sequence, and the mask sequence indicates the position of the non-text token in the first token sequence. Thus, the mask sequence can indicate the position of the non-text token, thereby enabling accurate identification of the non-text token in the first token sequence.

[0017] In a second aspect, a multimodal data inference apparatus is provided for performing operations of the first aspect or any possible implementation thereof.

[0018] In one possible implementation, the apparatus includes: a first acquisition module for acquiring a first token sequence of multimodal data, the first token sequence including non-text tokens; a second acquisition module for acquiring indicator parameters of non-text tokens based on a tail token, the tail token being located at the last position of the first token sequence and representing a feature of the multimodal data, the indicator parameters indicating the importance of the non-text tokens to the accuracy of the inference result of the multimodal data; a compression module for compressing the non-text tokens in the first token sequence according to the indicator parameters to obtain a second token sequence; and an inference module for performing inference based on the second token sequence to obtain the inference result of the multimodal data.

[0019] In one possible implementation, the second acquisition module is used to acquire a Q matrix generated based on the tail token and a K matrix generated based on the non-text token, wherein the Q matrix represents the information of interest in the inference process and the K matrix represents the features of the corresponding token; and the index parameters of the non-text token are acquired based on the similarity between the Q matrix and the K matrix.

[0020] In one possible implementation, the second acquisition module is used to use the similarity between the Q matrix and the K matrix as an indicator parameter of a non-text token; or, the similarity between the Q matrix and the K matrix is ​​pooled or smoothed, and the similarity between the processed Q matrix and the K matrix is ​​used as an indicator parameter of any non-text token.

[0021] In one possible implementation, the Q matrix is ​​a compressed Q matrix, where the number of attention heads in the uncompressed Q matrix is ​​greater than the number of attention heads in the K matrix, and the number of attention heads in the compressed Q matrix is ​​the same as the number of attention heads in the K matrix.

[0022] In one possible implementation, a first acquisition module is used to input multimodal data into a multimodal large model, which includes N layers of Transformers, where N is a positive integer; to acquire a first token sequence based on the output of the (m-1)th layer of Transformers, where m is a positive integer less than or equal to N; and an inference module is used to input a second token sequence into the mth layer of Transformers of the multimodal large model, and to acquire the inference result of the multimodal data based on the output of the multimodal large model.

[0023] In one possible implementation, the model variables in the Transformer at layer (m-1) and before layer (m-1) are matched with the size of the first token sequence, and the model variables in the Transformer at layer m and after layer m are matched with the size of the second token sequence.

[0024] In one possible implementation, a compression module is used to compress the non-text tokens in the first token sequence according to index parameters and a compression ratio to obtain a second token sequence, wherein the compression ratio is the ratio of the total number of non-text tokens in the second token sequence to the total number of non-text tokens in the first token sequence.

[0025] In one possible implementation, the values ​​of m and compression ratio are the values ​​of the multimodal large model under the condition that the search conditions are met: the inference accuracy of the multimodal large model is greater than the accuracy threshold, and the inference time of the multimodal large model is less than the time threshold.

[0026] In one possible implementation, the apparatus further includes: an identification module, configured to identify non-text tokens in the first token sequence based on a mask sequence corresponding to the first token sequence, wherein the length of the mask sequence is the same as the length of the first token sequence, and the mask sequence indicates the position of the non-text token in the first token sequence.

[0027] Thirdly, a multimodal data inference device is provided, comprising: a processor coupled to a memory, the memory storing at least one program instruction or code, the at least one program instruction or code being loaded and executed by the processor to enable the multimodal data inference device to implement the multimodal data inference method as described in the first aspect or any one of the first aspects.

[0028] Optionally, the processor may be one or more, and the memory may be one or more.

[0029] Optionally, the memory may be integrated with the processor, or the memory may be separated from the processor.

[0030] In the specific implementation process, the memory can be a non-transitory memory, such as read-only memory (ROM), which can be integrated with the processor on the same chip or set on different chips. This application does not limit the type of memory or the way the memory and processor are set.

[0031] Fourthly, a computer-readable storage medium is provided, the storage medium storing at least one instruction, the instruction being loaded and executed by a processor to cause a computer to implement the method described in the first aspect or any possible implementation of the first aspect.

[0032] Fifthly, a computer program (product) is provided, the computer program (product) comprising: computer program code, which, when executed by a computer, causes the computer to perform the methods described in the above aspects.

[0033] In a sixth aspect, a chip is provided, including a processor for retrieving and executing instructions stored in a memory, causing a communication device on which the chip is mounted to perform the methods described in the preceding aspects.

[0034] In a seventh aspect, another chip is provided, comprising: an input interface, an output interface, a processor, and a memory, wherein the input interface, the output interface, the processor, and the memory are connected via an internal connection path, and the processor is used to execute code in the memory, wherein when the code is executed, the processor is used to perform the methods in the foregoing aspects.

[0035] It should be understood that the beneficial effects achieved by the technical solutions of the second to seventh aspects of this application and their corresponding possible implementations can be found in the technical effects of the first aspect and its corresponding possible implementations described above, and will not be repeated here. Furthermore, the multimodal data inference device mentioned in the second aspect can be the chip mentioned in the sixth or seventh aspect, or the multimodal data inference device can also be the multimodal data inference device mentioned in the third aspect. Attached Figure Description

[0036] Figure 1 A schematic diagram of an implementation environment provided for an embodiment of this application;

[0037] Figure 2 A flowchart illustrating a method for reasoning multimodal data provided in this application embodiment;

[0038] Figure 3 This is a schematic diagram of the structure of a multimodal large model provided in an embodiment of this application;

[0039] Figure 4 A schematic diagram illustrating a hyperparameter search provided in an embodiment of this application;

[0040] Figure 5 A schematic diagram of the structure of a multimodal data inference device provided in an embodiment of this application;

[0041] Figure 6 This is a schematic diagram of the structure of a multimodal data inference device provided in an embodiment of this application. Detailed Implementation

[0042] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0043] With the development of AI technology, various fields such as smart homes, smart education, cultural entertainment, and finance are increasingly relying on multimodal data for reasoning. Multimodal data reasoning refers to the process of logical reasoning and decision-making by integrating data from multiple modalities (such as text, audio, images, and video). By fully utilizing the unique information contained in each modality, problems and scenarios can be described in a more comprehensive and richer way, thereby improving the accuracy and reliability of reasoning. For example, attention mechanisms can be used to dynamically allocate importance weights to different modalities in the reasoning process, thus better utilizing multimodal information. Taking multimodal tasks involving images and text as an example, attention mechanisms can automatically focus on areas in the image that are relevant to the text, as well as key parts of the text that correspond to the image content, based on specific task requirements, thereby enabling more accurate reasoning.

[0044] In the process of reasoning with multimodal data, the multimodal data can be converted into a series of tokens, and then reasoning can be performed on these multimodal tokens. A token is the smallest unit of information obtained after processing different modal data. It can transform high-dimensional multimodal data into low-dimensional discrete data, which is often more efficient for computers when processing discrete data. A multimodal token refers to a token obtained from multimodal data, including but not limited to at least two of the following: text tokens, image tokens, audio tokens, or video tokens. For example, text data can be converted into text tokens through word segmentation; image data can be converted into image tokens through a visual encoder, or an image can be divided into several fixed-size patches, each patch being considered an image token; video data can be obtained by tokenizing each frame of the video, or by cutting video segments into smaller segments as video tokens.

[0045] As the proportion of non-textual modal data in multimodal data continues to increase, the number of non-textual tokens derived from non-textual modal data is also increasing. A greater number of tokens leads to higher computational complexity during inference, resulting in slower inference speeds. Therefore, accelerating the inference process for multimodal data is a pressing issue that needs to be addressed.

[0046] This application provides a method for reasoning with multimodal data, which can compress non-text tokens according to their importance to reduce the number of tokens, thereby reducing computational complexity and improving reasoning speed. This method can be applied to any multimodal data reasoning scenario, including but not limited to long video understanding, audio understanding, and multimodal data hybrid understanding. The multimodal data includes non-text data, which includes at least one of image data, video data, and audio data. Optionally, the multimodal data also includes text data.

[0047] See Figure 1 , Figure 1 This diagram illustrates an implementation environment provided by an embodiment of this application. The implementation environment includes a terminal 11 and a server 12. Exemplarily, the terminal 11 has an application or webpage installed with multimodal data inference capabilities, and the server 12 provides background services for the application or webpage installed on the terminal.

[0048] The multimodal data inference method provided in this application embodiment can be executed by terminal 11, server 12, or jointly by terminal 11 and server 12; this application embodiment does not limit this. In the case where the method provided in this application embodiment is jointly executed by terminal 11 and server 12, server 12 undertakes the main computational work, and terminal 11 undertakes the secondary computational work; or, server 12 undertakes the secondary computational work, and terminal 11 undertakes the main computational work; or, server 12 and terminal 11 use a distributed computing architecture for collaborative computation.

[0049] Optionally, terminal 11 can be any electronic product capable of human-computer interaction with a user through one or more methods such as a keyboard, touchpad, touchscreen, remote control, voice interaction, or handwriting device, including personal computers (PCs), mobile phones, smartphones, personal digital assistants (PDAs), wearable devices, pocket PCs (PPCs), tablets, smart car systems, mobile robots, smart voice interaction devices, smart home appliances, in-vehicle terminals, virtual reality (VR) devices, and augmented reality (AR) devices. Server 12 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. Terminal 11 and server 12 establish a communication connection through wired or wireless networks.

[0050] Those skilled in the art should understand that the above-described terminal 11 and server 12 are merely examples. Other existing or future terminals or servers that are applicable to this application should also be included within the scope of protection of this application, and are hereby incorporated by reference.

[0051] See Figure 2 , Figure 2 This is a flowchart illustrating a method for inferring multimodal data provided in an embodiment of this application. The inference method for multimodal data is executed by a computing device, which can be... Figure 1 The terminal 11 shown can also be Figure 1 The server 12 shown in this embodiment is not limited thereto. Figure 2 As shown, the reasoning method for this multimodal data includes, but is not limited to, the following steps 201 to 204.

[0052] Step 201: Obtain the first token sequence of multimodal data, which includes non-text tokens.

[0053] Here, non-text tokens refer to tokens derived from non-text data in the multimodal data, including but not limited to at least one of image tokens, video tokens, or audio tokens. Optionally, if the multimodal data also includes text data, the first token sequence also includes text tokens, which are tokens derived from text data in the multimodal data. A token corresponds to a portion of any modality in the multimodal data, such as an image patch or an audio segment. For example, in the token sequence, each token is represented as a feature vector of the same dimension, which is obtained by extracting features from the corresponding token. For example, the first token sequence can be an n×d matrix, where n is the number of tokens and d is the feature dimension, with each row vector corresponding to one token.

[0054] This application does not limit the method of obtaining the first token sequence; it can be obtained using a modality-corresponding feature extraction model. For example, an image feature extraction model can be used to obtain the feature vectors corresponding to each token in the image data, and an audio feature extraction model can be used to obtain the feature vectors corresponding to each token in the audio data. This application does not limit the arrangement of the tokens in the first token sequence for multimodal data. Optionally, tokens of the same modality can be arranged together, with non-text tokens placed in the middle and text tokens placed at the beginning and end.

[0055] In one possible implementation, in addition to obtaining the first token sequence, a mask sequence for the first token sequence is also obtained. The mask sequence has the same length as the first token sequence and indicates the positions of text tokens and non-text tokens within the first token sequence. For example, a binary mask sequence with the same length as the first token sequence is created, setting the positions corresponding to non-text tokens to 1 and the positions corresponding to text tokens to 0. In this case, the method further includes: identifying non-text tokens in the first token sequence based on the mask sequence corresponding to the first token sequence.

[0056] Step 202: Obtain the indicator parameters of the non-text token based on the tail token. The tail token is located at the last position of the first token sequence and represents the characteristics of the multimodal data. The indicator parameters indicate the importance of the non-text token to the accuracy of the inference results of the multimodal data.

[0057] In this embodiment, when obtaining the feature vector corresponding to the tail token of the first token sequence, information from all preceding tokens is considered. Optionally, features of preceding tokens are adaptively fused based on the correlation between the tail token and all preceding tokens. For example, the greater the correlation between preceding and tail tokens, the greater the fusion weight; the smaller the correlation, the smaller the fusion weight. Thus, the tail token can integrate information from all preceding tokens, enabling it to represent the characteristics of multimodal data. Furthermore, the indicator parameters obtained from the tail token take into account the influence of the importance of text tokens on non-text tokens, improving the accuracy of the indicator parameters.

[0058] In one possible scenario, the indicator parameter can also be referred to as an importance score. The larger the indicator parameter of any non-text token, the greater its influence on the inference result; that is, the more important it is for accurate inference of multimodal data. Conversely, the smaller the indicator parameter of any non-text token, the smaller its influence on the inference result; that is, the less important it is for accurate inference of multimodal data. Therefore, deleting non-text tokens with smaller indicator parameters from the first token sequence will not significantly affect the final inference result and can reduce the computational complexity of the inference process, thereby shortening the inference time. The criterion for determining whether an indicator parameter is small can be flexibly configured. For example, a parameter threshold can be set; if the indicator parameter is less than the threshold, it is considered small. Alternatively, a sorting threshold can be set; if the indicator parameters are sorted from largest to smallest, those after the sorting threshold are considered small; if the indicator parameters are sorted from smallest to largest, those before the sorting threshold are considered small.

[0059] The embodiments of this application do not limit the method for obtaining the indicator parameters of non-text tokens based on the tail token. Optionally, for each non-text token in the first token sequence, the indicator parameters of each non-text token are obtained based on the similarity or correlation between each non-text token and the tail token.

[0060] For example, the similarity between any non-text token and the tail token can be directly used as the metric parameter for that non-text token; alternatively, pooling or smoothing can be applied to the similarity between each non-text token and the tail token, and the processed similarity between any non-text token and the tail token can be used as the metric parameter for that non-text token. Pooling is used to combine the similarity of local tokens into a representative value by taking the maximum or average value, thereby reducing the computational cost of similarity data. Smoothing is used to perform weighted averaging or other similar operations on the similarity data to reduce noise and fluctuations, making the similarity data smoother. The similarity data includes the similarity between each non-text token and the tail token.

[0061] In one possible implementation, the method for obtaining the indicator parameters of non-text tokens based on the tail token includes: obtaining a Q-matrix generated based on the tail token and a K-matrix generated based on the non-text tokens, where the Q-matrix represents the information of interest in the inference process and the K-matrix represents the features of the corresponding token; and obtaining the indicator parameters of the non-text tokens based on the similarity between the Q-matrix and the K-matrix. Each non-text token corresponds to one K-matrix. For multiple non-text tokens included in the first token sequence, the K-matrix corresponding to each non-text token is obtained, and then the similarity between the Q-matrix and the K-matrix corresponding to each non-text token is obtained. The method for generating the Q-matrix based on the tail token and the method for generating the K-matrix based on the non-text tokens are not limited in this embodiment.

[0062] Optionally, in the attention mechanism, the first token sequence can be transformed into its Q-matrix and K-matrix using two linear transformations, each corresponding to a different weight matrix. These weight matrices can be learned through model training. If the first token sequence includes the tail token, then the Q-matrix of the first token sequence includes the Q-matrix of the tail token. If the first token sequence includes non-text tokens, then the K-matrix of the first token sequence includes the K-matrix of the non-text tokens. The Q-matrix is ​​used to calculate similarity with the K-matrix to determine the relevance between the current token and other tokens in the token sequence. The K-matrix represents the features of each token in the token sequence and is used to calculate similarity with the Q-matrix.

[0063] Therefore, the similarity between non-text tokens and tail tokens can be accurately and quickly calculated using the Q and K matrices. A higher similarity to the tail token indicates higher importance, while a lower similarity indicates lower importance. This allows for the rapid acquisition of non-text token metrics based on similarity. Optionally, the Q and K matrices are used to calculate attention scores in the attention mechanism. When this method relies on attention mechanisms for inference, the Q and K matrices output by the attention operator can be directly used without separate calculation, thus accelerating the acquisition of metrics.

[0064] Optionally, the similarity between the Q matrix and the K matrix is ​​used to obtain the indicator parameters for non-text tokens. This includes: using the similarity between the Q matrix and the K matrix as the indicator parameter for non-text tokens; or, performing pooling or smoothing processing on the similarity between the Q matrix and the K matrix, and using the processed similarity between the Q matrix and the K matrix as the indicator parameter for any non-text token. Therefore, similarity can be used directly as the indicator parameter, or a variant of similarity can be used, improving the flexibility in obtaining the indicator parameter. The variant of similarity refers to the similarity after pooling or smoothing processing.

[0065] In multi-head attention mechanisms, the Q-matrix and K-matrix can each be divided into h attention heads for parallel computation, where h is a positive integer. Each attention head focuses on different aspects of the token sequence, thereby capturing richer information. When the number of attention heads in the Q-matrix differs from the number of attention heads in the K-matrix, appropriate compression processing is required to ensure that the Q-matrix of the tail token has the same number of attention heads as the K-matrix of any non-text token.

[0066] In one possible scenario, the Q-matrix of the tail token is the compressed Q-matrix. Before compression, the number of attention heads in the Q-matrix is ​​greater than the number of attention heads in the K-matrix. After compression, the number of attention heads in the Q-matrix is ​​the same as the number of attention heads in the K-matrix. This ensures that the Q-matrix of the tail token has the same number of attention heads as the K-matrix of the non-text token, facilitating the calculation of the similarity between the Q-matrix and the K-matrix and improving the efficiency of similarity calculation. For example, if the size of the Q-matrix of the tail token before compression is 1×h1×d, and the size of the K-matrix of any non-text token is 1×h2×d, then the size of the Q-matrix of the tail token after compression is 1×h2×d. Here, h1 represents the number of attention heads in the Q-matrix of the tail token, h2 represents the number of attention heads in the K-matrix of any non-text token, and d represents the feature dimension corresponding to each number of attention heads.

[0067] Step 203: Compress the non-text tokens in the first token sequence according to the index parameters to obtain the second token sequence.

[0068] After obtaining the indicator parameters of non-text tokens, the importance of non-text tokens to the inference result can be determined, thus identifying which non-text tokens in the first token sequence are important and which are not. For example, the top K non-text tokens with the largest indicator parameters in the first token sequence can be considered important non-text tokens, and the remaining non-text tokens can be considered unimportant non-text tokens, where K is a positive integer; alternatively, non-text tokens with indicator parameters greater than a first threshold in the first token sequence can be considered important non-text tokens, and those less than the first threshold can be considered unimportant non-text tokens, where the first threshold can be flexibly set; or, the non-text tokens in the first token sequence can be compressed according to a compression ratio. The compression ratio is the ratio of the total number of non-text tokens in the second token sequence to the total number of non-text tokens in the first token sequence, and the compression ratio can be represented by r.

[0069] Optionally, unimportant non-text tokens in the first token sequence are discarded, while important non-text tokens are retained to obtain a second token sequence. The total number of non-text tokens in the second token sequence is less than the total number of non-text tokens in the first token sequence, thus compressing the first token sequence. This compression can also be called pruning, i.e., removing unimportant non-text tokens. Since only non-text tokens unimportant to the inference result are discarded in the second token sequence, if the accuracy of the inference result based on the second token sequence is a first accuracy, and the accuracy of the inference result based on the first token sequence is a second accuracy, then the difference between the first accuracy and the second accuracy is less than a difference threshold. The difference threshold can be flexibly defined, and is usually a small number.

[0070] Step 204: Perform inference based on the second token sequence to obtain the inference results of the multimodal data.

[0071] This application does not limit the reasoning method based on the second token sequence; any reasoning method based on the first token sequence can be used. This application reduces the length of the second token sequence by compressing the first token sequence and the number of non-text tokens. The shorter the length of the second token sequence, the lower the complexity of reasoning based on it, thereby accelerating inference for multimodal data.

[0072] The indicator parameters indicating the importance of non-text tokens are obtained from tail tokens. Since tail tokens represent the characteristics of multimodal data, they incorporate the overall information of the multimodal data. Therefore, the indicator parameters obtained from tail tokens take into account the impact of multimodality on non-text tokens, improving the accuracy of the indicator parameters for non-text tokens, and thus improving the accuracy of compressing the number of non-text tokens according to the indicator parameters. Furthermore, compared to methods that determine the importance of non-text tokens based on multimodal data, this method determines the importance of non-text tokens only based on tail tokens. The amount of data in tail tokens is smaller than that in multimodal data, reducing the computational burden of determining the importance of non-text tokens and further accelerating the inference process for multimodal data. Compared to methods that determine the importance of non-text tokens based on attention scores provided by attention operators, this method does not rely on attention scores, making it adaptable to attention operators that cannot provide attention scores.

[0073] The following section will introduce the inference method for multimodal data and its combination with a large multimodal model, as provided in the embodiments of this application.

[0074] Large models refer to AI models with a large number of parameters and complex computational structures. They are typically built from deep neural networks and have billions or even hundreds of billions of parameters. AI models are a class of mathematical algorithm models that use machine learning concepts to solve practical problems. Multimodal large models are generative large models capable of processing and fusing multiple modalities of data (such as text, images, audio, and video). Generative large models are used to perform inference based on a given inference request and generate inference results. For example, an inference request might instruct inference based on text, images, audio, or video to generate inference results.

[0075] Taking a multimodal large-scale model as an example of an AI model based on the Transformer architecture, this multimodal large-scale model typically includes an input layer, multiple Transformer layers, and an output layer. The Transformer layers are also called Transformer blocks. The input layer is used to convert the input multimodal data into a token sequence, for example, a token sequence X∈R. n × dHere, n is the length of the token sequence (number of tokens), and d is the dimension of the input features. A Transformer block refers to the basic building block in a Transformer model. Multiple Transformer blocks are stacked sequentially to form a complete Transformer model, enabling effective feature extraction and representation learning of the token sequence. The output layer converts the final output of the Transformer layers into the model's output.

[0076] For example, the Transformer layer includes components such as multi-head self-attention, residual connection, layer normalization, and feed-forward network (FFN).

[0077] Multi-head self-attention is a core component of the Transformer architecture. It captures contextual information by calculating the correlations between different positions in a token sequence. Multi-head self-attention allows the model to learn contextual information about the token sequence (such as words, letters, image features, etc.) in parallel from different representation subspaces, thereby enhancing the model's expressive power. This can be referred to as the attention computation task of the Transformer layer. The structure of multi-head self-attention mainly includes the following parts: query-key-value (QKV) matrix generation, multi-head attention layer, and linear projection (Projection).

[0078] QKV matrix generation refers to transforming the token sequence X∈R using three linear transformations. n × d This is transformed into Q, K, and value (V) matrices. The three linear transformations correspond to different weight matrices, denoted as W. Q W K and W V The Q matrix is ​​used to calculate similarity with the K matrix to determine the relevance between the current token and other tokens in the token sequence. The K matrix represents the features of each token in the token sequence and is used to calculate similarity with the Q matrix. The V matrix is ​​the information carrier of each token in the token sequence, storing the specific information of the token.

[0079] A multi-head attention layer, or simply an attention layer, refers to dividing the Q, K, and V matrices into h attention heads for parallel computation (h is a positive integer). Each attention head focuses on a different aspect of the token sequence, thereby capturing richer information. The computational characteristic of an attention layer is bandwidth-intensive. For example, by applying different linear transformations (matrix multiplication) to the Q, K, and V matrices, they are projected onto h different subspaces (i.e., different heads). Taking attention head i as an example... W i Q W i K W i V Let d be the projection matrix of attention head i, where i is a positive integer less than or equal to h and greater than 0. k =d / h.

[0080] For any attention head, calculate the attention score of attention head i. (Also about to Q) i Matrix and K i Multiplying the transposes of matrices has a time complexity of O(n^2). 2 ×d k Normalize the attention score, for example, through the Softmax normalization operation, denoted as Softmax(S). i ), S i The attention score representing attention head i has a time complexity of O(n^2). 2 The normalized result is compared with V. i Matrix multiplication yields the attention output of attention head i, also known as the context vector. The time complexity is O(n 2 ×d k Therefore, the total time complexity of the multi-head self-attention mechanism is O(n^2). 2 ×d k +n 2 +n 2 ×d k )≈O(n 2 ×d), when d is a constant, the time complexity mainly depends on the number of tokens n in the token sequence, that is, the time complexity of the multi-head self-attention mechanism is quadratic with the number of tokens.

[0081] Linear projection refers to concatenating the attention outputs of multiple attention heads and then performing a linear projection on the concatenated matrix to obtain the output result of the multi-head self-attention mechanism. The weight matrix corresponding to linear projection can be represented as W. O .

[0082] After the output result is obtained through the multi-head self-attention mechanism described above, it is sequentially processed through residual connections and layer normalization before being input into the FFN. The FFN consists of two linear layers (such as FF1 and FF2) and a non-linear activation function (usually the ReLU function). The FFN is used to further transform features and extract information from the layer-normalized output, enhancing the model's expressive power. A residual connection is then added to the output of the FFN, which is then added to the previously layer-normalized input, followed by another layer normalization operation to obtain the output result of the Transformer layer. The Transformer layer output includes the token sequence output by the Transformer layer and the corresponding QKV matrix. Optionally, the Transformer layer output also includes the attention score of each token.

[0083] Optionally, the attention operators used in the Transformer layer include, but are not limited to, multi-head attention, self-attention, flash attention, scaled dot product attention (SDPA), or paged attention. Flash attention is a fast attention calculation technique, while paged attention is a technique that performs attention calculation using a paginated key-value cache. When using Flash attention, SDPA, or paged attention, the output of each Transformer layer does not include an attention score, but it does include a QKV matrix.

[0084] Furthermore, the inference process of a multimodal large model typically includes two inference stages: a prefill stage and a decoding stage. The Transformer layers in both the prefill and decoding stages can be the same, both including the components described above. The prefill stage is also called the full inference stage, and the decoding stage is also called the incremental inference stage. The prefill stage is used to process the token sequence of the multimodal large model based on a multi-head self-attention mechanism, generating a corresponding KV matrix for attention calculation. At the end of the prefill stage, the multimodal large model generates its first output token and establishes a KV cache for use in subsequent decoding stages. The KV cache contains the KV matrix of the token sequence.

[0085] The decoding phase generates the next output token based on information such as the KV matrix in the KV cache, without reprocessing the entire token sequence. As each output token is generated, the KV cache is updated accordingly, thus progressively generating all output tokens, i.e., the output inference result. The decoding phase is an iterative process, generating a new token in each iteration.

[0086] In summary, multimodal large models enable inference on multimodal data. For example, multimodal data is input into a multimodal large model, and inference is performed on the multimodal data through the multimodal large model. The output of the multimodal large model is the inference result of the multimodal data. Furthermore, the inference method provided in this application embodiment can be implemented in conjunction with a multimodal large model, or in other words, the inference method provided in this application embodiment can optimize the multimodal large model to reduce the computational complexity in the multimodal large model inference process and shorten the inference time of the multimodal large model.

[0087] In one possible implementation, obtaining the first token sequence of multimodal data includes: inputting the multimodal data into a large multimodal model, which includes N Transformer layers, where N is a positive integer; and obtaining the first token sequence based on the output of the (m-1)th Transformer layer, where m is a positive integer less than or equal to N. In this case, inference based on the second token sequence to obtain the inference result of the multimodal data includes: inputting the second token sequence into the m-th Transformer layer of the large multimodal model; and obtaining the inference result of the multimodal data based on the output of the large multimodal model. Thus, this method can combine a large multimodal model to perform inference on multimodal data, thereby accelerating the inference time of the large multimodal model.

[0088] For example, Figure 3 This is a schematic diagram of the structure of a multimodal large model provided in an embodiment of this application. For example... Figure 3As shown, an importance evaluation module is included between the (m-1)th and m-th Transformers. This module obtains the importance score of each non-text token in the first token sequence based on the tail token. The importance score corresponds to the aforementioned index parameters. For example, the importance evaluation module: obtains the first token sequence, the Q matrix of the tail token, and the K matrix of each non-text sequence based on the output of the (m-1)th Transformer; compresses the number of attention heads in the Q matrix to match the number of attention heads in the K matrix; then obtains the similarity between the compressed Q matrix and the K matrix; and finally determines the importance score based on the similarity. The similarity can be calculated by performing matrix multiplication on the compressed Q matrix and the K matrix to obtain the operation vector; then calculating the similarity using the Softmax function.

[0089] The Softmax function transforms the input numerical vector into a probability distribution. In this distribution, elements with higher probability values ​​are, in a sense, more similar and therefore more important. Optionally, a low-precision Softmax function can be used. This reduces computation and storage space by quantizing the input data or using low-precision data types (such as half-precision floating-point numbers, integers, etc.) to represent and compute the Softmax function, while maintaining model performance as much as possible. Figure 3 As shown, the Softmax function of fp16 can be used. fp16 is a half-precision floating-point data type, that is, it uses 16 bits to represent a floating-point number, which can reduce the space and time occupied by data storage and calculation.

[0090] In this importance assessment module, the importance score of each non-text token is obtained only based on the tail token. Since the tail token has less data than the first token sequence, the computational cost of determining the importance score of non-text tokens is reduced, further accelerating the inference speed of multimodal large models. Furthermore, this importance assessment module does not rely on the attention score output by the Transformer layer and can be widely adapted to various attention operators, such as FlashAttention, SDPA, and PagedAttention, which do not return attention scores.

[0091] like Figure 3As shown, a non-text pruning module is included between the (m-1)th Transformer layer and the mth Transformer layer. This module compresses the non-text tokens in the first token sequence according to their importance scores to obtain the second token sequence. When the non-text tokens are visual tokens, the non-text pruning module becomes the visual pruning module. The visual pruning module compresses the visual tokens in the first token sequence according to their importance scores to obtain the second token sequence.

[0092] In the non-text pruning module, for the first token sequence output by the (m-1)th layer Transformer, the non-text tokens in the first token sequence are pruned according to the importance score output by the importance evaluation module. The non-text tokens with the highest importance score (r%) are retained, and the pruned second token sequence is input into the m-th layer Transformer. For example... Figure 3 As shown, black boxes represent text tokens, and white boxes represent non-text tokens. After pruning, non-text tokens with low importance scores are removed.

[0093] Because the token sequence is compressed between the (m-1)th and (m)th Transformers in the multimodal large model, the token sequences processed by the (m-1)th and earlier Transformers have different lengths than those processed by the (m)th and later Transformers. In this embodiment, the model variables in the (m-1)th and earlier Transformers match the size of the first token sequence, and the model variables in the (m)th and later Transformers match the size of the second token sequence. Here, model variables refer to those related to the token sequence. Therefore, in scenarios applying multimodal large models, the model variables change with the length of the token sequence.

[0094] For example, after the non-text pruning module obtains the second token sequence, the model variables related to the token sequence in the multimodal large model are updated according to the length of the second token sequence. Optionally, the model variables include, but are not limited to, mask sequences, KV caches, etc. Thus, in the pre-filling stage, by pruning the token sequence, the number of tokens is reduced, thereby reducing the inference computation in the pre-filling stage; in the encoding stage, inference is performed based on the KV matrix. Since the KV matrix is ​​generated based on the token sequence, the reduction in the number of tokens will lead to a reduction in the KV matrix, which will also reduce the computation in the encoding stage. That is, by reducing the number of non-text tokens, this embodiment of the application can reduce the computation in both the pre-filling and encoding stages, thereby accelerating the inference in both stages.

[0095] In this embodiment, the number of pruning layers *m* and the compression ratio *r* can be considered as model parameters of the multimodal large model. Optionally, the values ​​of *m* and *r* are the values ​​of the multimodal large model under the condition that the search criteria are met: the inference accuracy of the multimodal large model is greater than an accuracy threshold, and the inference time of the multimodal large model is less than a time threshold. Both the accuracy threshold and the time threshold can be set empirically or flexibly adjusted according to the application scenario. Therefore, by performing inference with the values ​​of *m* and *r* that meet the search criteria, the multimodal large model can ensure that the inference accuracy is greater than the accuracy threshold and the inference time is less than the time threshold.

[0096] For example, the optimal values ​​of parameters m and r can be solved using hyperparameter search, such as through a Bayesian optimizer. The basic principle of a Bayesian optimizer is to construct a probabilistic model of the hyperparameters and objective function values ​​based on previous evaluation results, and then use this model to predict the next set of hyperparameters most likely to improve the objective function value. For instance, a set of hyperparameters is randomly selected for evaluation to obtain the corresponding objective function values; based on the existing hyperparameters and objective function values, a probabilistic model is constructed using methods such as Gaussian processes. This model estimates the distribution of the objective function values ​​under different combinations of hyperparameters; based on the probabilistic model, a collection function (such as expected improvement, confidence upper bound, etc.) is used to select the next set of potentially promising hyperparameters for evaluation; the newly evaluated hyperparameters and objective function values ​​are added to the existing data to update the probabilistic model; this process continues until a preset number of evaluations is reached or other termination conditions are met. The collection function balances exploration (trying new combinations of hyperparameters) and utilization (selecting combinations near known well-performing hyperparameters).

[0097] See Figure 4The diagram illustrates the hyperparameter search. In the multimodal large model provided in this application embodiment, the hyperparameters include the number of pruning layers *m* and the compression ratio *r*. The objective function is to minimize the inference time while ensuring the accuracy of the inference results on the test set is greater than 0.9, i.e., min T(m,r), Subject to f(m,r)>0.9. The test set is a dataset including multimodal data. The search range for the number of pruning layers *m* is a positive integer from 1 to N, and the range for the compression ratio *r* is a real number from 0 to 1. For example, different combinations of *m* and *r* are configured in the multimodal large model. The multimodal large model is used to perform inference on the test set, and the accuracy of the inference results under different combinations of *m* and *r* is observed. Gradually, the values ​​of *m* and *r* that minimize the inference time *T* of the multimodal large model while ensuring the inference accuracy on the test set are found.

[0098] In summary, the computational complexity of multimodal large models increases dramatically when handling inference tasks with long token sequences. This application's embodiments address this by calculating the importance score of non-text tokens based on the tail token between Transformer layers of the multimodal large model. This fully integrates textual information, achieving effective compression of visual non-text tokens and significantly improving the inference speed of the multimodal large model.

[0099] An example is provided to illustrate a scenario where the inference method for multimodal data provided in this application is applied. In the medical field, multimodal data includes data from various modalities such as medical images, patient medical records, and symptom descriptions. The method provided in this application obtains a first token sequence of multimodal data. The first token sequence includes tokens converted from data from various modalities such as medical images, patient medical records, and symptom descriptions. The tokens converted from medical images are non-text tokens, while the tokens converted from patient medical records and symptom descriptions are text tokens. Compression processing is applied to the non-text tokens based on indicator parameters, i.e., compression processing is applied to medical images. This reduces the computational load during the inference process, thereby shortening the inference time.

[0100] When this method is combined with a multimodal large model, the model structure of the multimodal large model is as follows: Figure 3 As shown, the multimodal large model takes multimodal data such as medical images, patient medical records, and symptom descriptions as input, and outputs inference results such as disease type and severity to assist doctors in medical image diagnosis. During the inference process on multimodal data such as medical images, patient medical records, and symptom descriptions, the multimodal large model compresses medical images through an importance assessment module and a non-text pruning module.

[0101] In the field of autonomous driving, multimodal data includes road images captured by vehicle cameras, historical driving data, and other data from various modalities. The method provided in this application provides a first token sequence for the multimodal data. This first token sequence includes tokens obtained by converting data from various modalities, such as road images and historical driving data. The tokens obtained from converting road images are non-text tokens, while the tokens obtained from converting historical driving data are text tokens. Compression processing is applied to the non-text tokens based on indicator parameters; this is equivalent to compression processing of the road images. This reduces the computational load during inference and thus shortens the inference time.

[0102] Similarly, in this method, combined Figure 3 In the case of the multimodal large model shown, the input to the multimodal large model consists of multimodal data such as road images captured by vehicle cameras and historical driving data. The output of the multimodal large model is the inference result, such as whether there are obstacles and the type of obstacles, enabling the autonomous driving system to take actions such as braking or avoidance in advance to ensure driving safety. During the inference process of multimodal data such as road images and historical driving data, the multimodal large model compresses the road images through an importance assessment module and a non-text pruning module.

[0103] Furthermore, multimodal data can also encompass various modalities of data in fields such as security monitoring, industrial manufacturing and quality inspection, agricultural production, intelligent robots, intelligent document and content creation, and intelligent education. The reasoning methods provided in this application's embodiments enable accelerated reasoning in these various fields. The implementation methods for the reasoning processes in different fields are described above, and will not be repeated in detail in this application's embodiments.

[0104] The above describes the inference method for multimodal data according to the embodiments of this application. Corresponding to the above method, the embodiments of this application also provide an inference device for multimodal data. Figure 5 This is a schematic diagram of the structure of a multimodal data inference device provided in an embodiment of this application. This device is capable of performing... Figure 2 All or part of the operations in the method shown. It should be understood that the device may include more additional modules than those shown or omit some of the modules shown, and the embodiments of this application do not limit this.

[0105] like Figure 5As shown, the device includes: a first acquisition module 501, used to acquire a first token sequence of multimodal data, the first token sequence including non-text tokens; a second acquisition module 502, used to acquire indicator parameters of non-text tokens based on the tail token, the tail token being located at the last position of the first token sequence, and the tail token representing the characteristics of the multimodal data, the indicator parameters indicating the importance of non-text tokens to the accuracy of the inference result of the multimodal data; a compression module 503, used to compress the non-text tokens in the first token sequence according to the indicator parameters to obtain a second token sequence; and an inference module 504, used to perform inference based on the second token sequence to obtain the inference result of the multimodal data.

[0106] In one possible implementation, the second acquisition module 502 is used to acquire a Q matrix generated based on the tail token and a K matrix generated based on the non-text token. The Q matrix represents the information of interest in the inference process, and the K matrix represents the features of the corresponding token. Based on the similarity between the Q matrix and the K matrix, the index parameters of the non-text token are acquired.

[0107] In one possible implementation, the second acquisition module 502 is used to use the similarity between the Q matrix and the K matrix as an indicator parameter of a non-text token; or, the similarity between the Q matrix and the K matrix is ​​pooled or smoothed, and the similarity between the processed Q matrix and the K matrix is ​​used as an indicator parameter of any non-text token.

[0108] In one possible implementation, the Q matrix is ​​a compressed Q matrix, where the number of attention heads in the uncompressed Q matrix is ​​greater than the number of attention heads in the K matrix, and the number of attention heads in the compressed Q matrix is ​​the same as the number of attention heads in the K matrix.

[0109] In one possible implementation, the first acquisition module 501 is used to input multimodal data into a multimodal large model, which includes N layers of Transformers, where N is a positive integer; to obtain a first token sequence based on the output of the (m-1)th layer Transformer, where m is a positive integer less than or equal to N; and the inference module 504 is used to input a second token sequence into the mth layer Transformer of the multimodal large model, and to obtain the inference result of the multimodal data based on the output of the multimodal large model.

[0110] In one possible implementation, the model variables in the Transformer at layer (m-1) and before layer (m-1) are matched with the size of the first token sequence, and the model variables in the Transformer at layer m and after layer m are matched with the size of the second token sequence.

[0111] In one possible implementation, compression module 503 is used to compress the non-text tokens in the first token sequence according to the index parameters and compression ratio to obtain a second token sequence, wherein the compression ratio is the ratio of the total number of non-text tokens in the second token sequence to the total number of non-text tokens in the first token sequence.

[0112] In one possible implementation, the values ​​of m and compression ratio are the values ​​of the multimodal large model under the condition that the search conditions are met: the inference accuracy of the multimodal large model is greater than the accuracy threshold, and the inference time of the multimodal large model is less than the time threshold.

[0113] In one possible implementation, the apparatus further includes: an identification module, configured to identify non-text tokens in the first token sequence based on a mask sequence corresponding to the first token sequence, wherein the length of the mask sequence is the same as the length of the first token sequence, and the mask sequence indicates the position of the non-text token in the first token sequence.

[0114] It should be understood that the above Figure 5 The provided device, when implementing its functions, is only illustrated by the above-described division of functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process and beneficial effects are detailed in the method embodiments, and will not be repeated here.

[0115] See Figure 6 , Figure 6 A schematic diagram of the structure of a multimodal data inference device 2000 provided in an exemplary embodiment of this application is shown. Figure 6 The multimodal data inference device 2000 shown is used to perform the above. Figure 2 The operations involved in the multimodal data inference method shown are illustrated. The multimodal data inference device 2000 is, for example, a computing device such as a terminal or server, and can be implemented using a general bus architecture. Figure 6As shown, the multimodal data inference device 2000 includes at least one processor 2001, a memory 2003, and at least one communication interface 2004.

[0116] Processor 2001 may be, for example, a general-purpose central processing unit (CPU), a digital signal processor (DSP), a network processor (NP), a graphics processing unit (GPU), a neural-network processing unit (NPU), a data processing unit (DPU), a microprocessor, or one or more integrated circuits for implementing the embodiments of this application. For example, processor 2001 includes application-specific integrated circuits (ASICs), programmable logic devices (PLDs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A PLD may be, for example, a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), generic array logic (GAL), or any combination thereof. It can implement or execute the various logic blocks, modules, and circuits described in connection with the embodiments of this invention. A processor may also be a combination that implements computational functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0117] Optionally, the multimodal data inference device 2000 also includes a bus. The bus is used to transfer information between the components of the multimodal data inference device 2000. The bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The symbol is represented by only one line, but this does not mean that there is only one bus or one type of bus.

[0118] Memory 2003 may be, for example, read-only memory (ROM) or other types of static storage devices capable of storing static information and instructions; random access memory (RAM) or other types of dynamic storage devices capable of storing information and instructions; electrically erasable programmable read-only memory (EEPROM); compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.); magnetic disk storage media or other magnetic storage devices; or any other medium capable of carrying or storing desired program code in the form of instructions or data structures and accessible by a computer, but not limited thereto. Memory 2003 may exist independently and be connected to processor 2001 via a bus. Memory 2003 may also be integrated with processor 2001.

[0119] The communication interface 2004 uses any transceiver-like device for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), or wireless local area networks (WLAN). The communication interface 2004 can include wired and wireless communication interfaces. Specifically, the communication interface 2004 can be an Ethernet interface, a Fast Ethernet (FE) interface, a Gigabit Ethernet (GE) interface, an Asynchronous Transfer Mode (ATM) interface, a WLAN interface, a cellular network communication interface, or a combination thereof. The Ethernet interface can be an optical interface, an electrical interface, or a combination thereof. In this embodiment, the communication interface 2004 can be used by the multimodal data inference device 2000 to communicate with other devices.

[0120] In a specific implementation, as one example, the processor 2001 may include one or more CPUs, such as... Figure 6The CPUs shown are CPU0 and CPU1. Each of these processors can be a single-core CPU or a multi-core CPU. A processor here can refer to one or more devices, circuits, and / or processing cores used to process data (e.g., computer program instructions).

[0121] In a specific implementation, as one example, the multimodal data inference device 2000 may include multiple processors, such as... Figure 6 The processors shown are 2001 and 2005. Each of these processors can be a single-core CPU or a multi-core CPU. Here, "processor" can refer to one or more devices, circuits, and / or processing cores used to process data (such as computer program instructions).

[0122] In a specific implementation, as one embodiment, the multimodal data inference device 2000 may further include an output device and an input device. The output device communicates with the processor 2001 and can display information in various ways. For example, the output device may be a liquid crystal display (LCD), a light-emitting diode (LED) display device, a cathode ray tube (CRT) display device, or a projector, etc. The input device communicates with the processor 2001 and can receive user input in various ways. For example, the input device may be a mouse, keyboard, touchscreen device, or sensing device, etc.

[0123] In some embodiments, the memory 2003 stores program code 2010 for executing the solution of this application, and the processor 2001 can execute the program code 2010 stored in the memory 2003. That is, the multimodal data inference device 2000 can implement the multimodal data inference method provided in the method embodiment through the processor 2001 and the program code 2010 in the memory 2003. The program code 2010 may include one or more software modules. Optionally, the processor 2001 itself may also store program code or instructions for executing the solution of this application.

[0124] In a specific embodiment, the multimodal data inference device 2000 of this application embodiment can correspond to the computing device in the above-described method embodiments. The processor 2001 in the multimodal data inference device 2000 reads instructions from the memory 2003, causing... Figure 6The multimodal data inference device 2000 shown is capable of performing all or part of the operations performed by the computing device.

[0125] In one possible implementation, the processor 2001 is used to acquire a first token sequence of multimodal data, the first token sequence including non-text tokens; acquire indicator parameters of non-text tokens based on the tail token, the tail token being located at the last position of the first token sequence and representing the characteristics of the multimodal data, the indicator parameters indicating the importance of non-text tokens to the accuracy of the inference result of the multimodal data; compress the non-text tokens in the first token sequence according to the indicator parameters to obtain a second token sequence; and perform inference based on the second token sequence to obtain the inference result of the multimodal data.

[0126] Other alternative implementation methods are described in the above method embodiments and will not be repeated here.

[0127] The multimodal data inference device 2000 can also correspond to the above. Figure 5 The multimodal data inference apparatus shown in the diagram is implemented using software from the multimodal data inference device 2000. In other words, the functional modules included in the multimodal data inference apparatus are generated by the processor 2001 of the multimodal data inference device 2000 reading the program code 2010 stored in the memory 2003.

[0128] in, Figure 2 Each step of the multimodal data inference method shown is completed through the integrated logic circuitry of the hardware or the instructions in the software form of the processor of the multimodal data inference device 2000. The steps of the method disclosed in the embodiments of this application can be directly implemented by the hardware processor, or by a combination of hardware and software modules in the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. Since the storage medium is located in memory, the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method; to avoid repetition, these will not be described in detail here.

[0129] This application also provides a server, which can vary significantly due to differences in configuration or performance. It may include one or more processors and one or more memories, wherein at least one computer program is stored in the one or more memories. This at least one computer program is loaded and executed by the one or more processors to enable the server to implement the multimodal data inference methods provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0130] It should be understood that the aforementioned processor can be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. General-purpose processors can be microprocessors or any conventional processor. It is worth noting that the processor can be a processor supporting Advanced Reduced Instruction Set Computing (RISC) machines (ARM) architecture.

[0131] Furthermore, in an alternative embodiment, the memory described above may include read-only memory and random access memory, and provide instructions and data to the processor. The memory may also include non-volatile random access memory. For example, the memory may also store device type information.

[0132] The memory can be volatile or non-volatile, or may include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which serves as an external cache. Many forms of RAM are available by way of example, but not limitation. Examples include static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0133] This application also provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to enable a computer to implement any of the above-described multimodal data inference methods.

[0134] This application also provides a computer program (product) that, when executed by a computer, causes the processor or computer to perform the corresponding steps and / or processes in the above method embodiments.

[0135] This application also provides a chip, including a processor, for calling and running instructions stored in a memory, causing a communication device equipped with the chip to execute any of the above-described multimodal data inference methods.

[0136] This application embodiment also provides another chip, including: an input interface, an output interface, a processor, and a memory. The input interface, output interface, processor, and memory are connected through an internal connection path. The processor is used to execute code in the memory. When the code is executed, the processor is used to execute any of the above-mentioned multimodal data inference methods.

[0137] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the multimodal data involved in this application were obtained with full authorization.

[0138] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product. A computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions according to this application are generated, in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk), etc.

[0139] Those skilled in the art will recognize that the method steps and modules described in conjunction with the embodiments disclosed herein can be implemented in software, hardware, firmware, or any combination thereof. To clearly illustrate the interchangeability of hardware and software, the steps and components of each embodiment have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0140] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0141] When implemented using software, it can be implemented wholly or partially as a computer program product. This computer program product includes one or more computer program instructions. As an example, the methods of this application embodiment can be described in the context of machine-executable instructions, such as program modules that execute on a device on a real or virtual processor of the target. Generally, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., which perform specific tasks or implement specific abstract data structures. In various embodiments, the functionality of program modules can be combined or divided among the described program modules. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside on both local and remote storage media.

[0142] Computer program code used to implement the methods of the embodiments of this application may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus, such that when executed by the computer or other programmable data processing apparatus, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a standalone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0143] In the context of the embodiments of this application, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, etc.

[0144] Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0145] A machine-readable medium can be any tangible medium that contains or stores programs for or relating to an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can include, but are not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. More detailed examples of machine-readable storage media include electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0146] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and modules described above can be found in the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0147] In the embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the couplings or direct couplings or communication connections shown or discussed may be indirect couplings or communication connections through some interfaces, devices, or modules, or they may be electrical, mechanical, or other forms of connection.

[0148] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of the embodiments of this application, depending on actual needs.

[0149] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0150] If the integrated module is implemented as a software functional module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0151] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items that have substantially the same function and purpose. It should be understood that there is no logical or temporal dependency between "first," "second," and "m," nor does it limit the quantity or order of execution. It should also be understood that although the following description uses the terms "first," "second," etc., to describe various elements, these elements should not be limited by the terms. These terms are merely used to distinguish one element from another. For example, without departing from the scope of various examples, a first image can be referred to as a second image, and similarly, a second image can be referred to as a first image. Both the first image and the second image can be images, and in some cases, they can be separate and distinct images.

[0152] It should also be understood that, in the various embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0153] In this application, the term "at least one" means one or more, and the term "multiple" means two or more. For example, multiple second messages refer to two or more second messages. The terms "system" and "network" are often used interchangeably in this document.

[0154] It should be understood that the terminology used in the description of the various examples herein is for the purpose of describing particular examples only and is not intended to be limiting. As used in the description of the various examples and the appended claims, the singular forms “a” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0155] It should also be understood that the term "and / or" as used herein refers to and covers any and all possible combinations of one or more of the associated listed items. The term "and / or" describes an association between related objects, indicating that three relationships can exist; for example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the character " / " in this application generally indicates that the preceding and following related objects are in an "or" relationship.

[0156] It should also be understood that the term “comprising” (also referred to as “includes”, “including”, “comprises” and / or “comprising”) as used in this specification specifies the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0157] It should also be understood that the terms “if” and “if” can be interpreted as meaning “when” or “upon”, or “in response to determination” or “in response to detection”. Similarly, depending on the context, the phrases “if determination…” or “if detection [the stated condition or event]” can be interpreted as meaning “when determination…”, or “in response to determination…”, or “when detection [the stated condition or event]” or “in response to detection [the stated condition or event]”.

[0158] It should be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0159] It should also be understood that the phrases "an embodiment," "an embodiment," and "a possible implementation" used throughout the specification mean that a specific feature, structure, or characteristic related to an embodiment or implementation is included in at least one embodiment of this application. Therefore, the phrases "in an embodiment," "an embodiment," or "a possible implementation" appearing throughout the specification do not necessarily refer to the same embodiment. Furthermore, these specific features, structures, or characteristics can be combined in any suitable manner in one or more embodiments.

[0160] The above description is only an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A reasoning method for multimodal data, characterized in that, The method includes: Obtain a first token sequence of multimodal data, the first token sequence including non-text tokens; The indicator parameters of the non-text token are obtained based on the tail token, which is located at the last position of the first token sequence and represents the characteristics of the multimodal data. The indicator parameters indicate the importance of the non-text token to the accuracy of the inference results of the multimodal data. The non-text tokens in the first token sequence are compressed according to the indicator parameters to obtain the second token sequence. Inference is performed based on the second token sequence to obtain the inference result of the multimodal data.

2. The method according to claim 1, characterized in that, The metrics parameters for obtaining the non-text token based on the tail token include: Obtain the query Q matrix generated based on the tail token and the key K matrix generated based on the non-text token, wherein the Q matrix represents the information of interest in the inference process and the K matrix represents the features of the corresponding token; Based on the similarity between the Q matrix and the K matrix, the indicator parameters of the non-text token are obtained.

3. The method according to claim 2, characterized in that, The step of obtaining the indicator parameters of the non-text token based on the similarity between the Q matrix and the K matrix includes: The similarity between the Q matrix and the K matrix is ​​used as the indicator parameter for the non-text token; Alternatively, pooling or smoothing can be applied to the similarity between the Q matrix and the K matrix, and the processed similarity between the Q matrix and the K matrix can be used as an indicator parameter for any non-text token.

4. The method according to claim 2 or 3, characterized in that, The Q matrix is ​​a compressed Q matrix. The number of attention heads in the uncompressed Q matrix is ​​greater than the number of attention heads in the K matrix. The number of attention heads in the compressed Q matrix is ​​the same as the number of attention heads in the K matrix.

5. The method according to any one of claims 1-4, characterized in that, The first token sequence for acquiring multimodal data includes: The multimodal data is input into a large multimodal model, which includes N Transformer layers, where N is a positive integer. The first token sequence is obtained based on the output of the (m-1)th layer Transformer, where m is a positive integer less than or equal to N; The reasoning based on the second token sequence to obtain the reasoning result of the multimodal data includes: The second token sequence is input into the m-th layer Transformer of the multimodal large model, and the inference result of the multimodal data is obtained based on the output of the multimodal large model.

6. The method according to claim 5, characterized in that, The model variables in the Transformer at layer (m-1) and before layer (m-1) match the size of the first token sequence, and the model variables in the Transformer at layer (m) and after layer (m) match the size of the second token sequence.

7. The method according to claim 5 or 6, characterized in that, The step of compressing the non-text tokens in the first token sequence according to the index parameters to obtain the second token sequence includes: The non-text tokens in the first token sequence are compressed according to the index parameters and compression ratio to obtain the second token sequence. The compression ratio is the ratio of the total number of non-text tokens in the second token sequence to the total number of non-text tokens in the first token sequence.

8. The method according to claim 7, characterized in that, The values ​​of m and the compression ratio are the values ​​of the multimodal large model under the condition that the search conditions are met. The search conditions are: the inference accuracy of the multimodal large model is greater than the accuracy threshold, and the inference time of the multimodal large model is less than the time threshold.

9. The method according to any one of claims 1-8, characterized in that, The method further includes: The non-text tokens in the first token sequence are identified based on the mask sequence corresponding to the first token sequence. The length of the mask sequence is the same as the length of the first token sequence, and the mask sequence indicates the position of the non-text token in the first token sequence.

10. A multimodal data inference device, characterized in that, The device includes: The first acquisition module is used to acquire a first token sequence of multimodal data, wherein the first token sequence includes non-text tokens; The second acquisition module is used to acquire the indicator parameters of the non-text token based on the tail token, wherein the tail token is located at the last position of the first token sequence, and the tail token represents the characteristics of the multimodal data, and the indicator parameters indicate the importance of the non-text token to the accuracy of the inference result of the multimodal data; The compression module is used to compress the non-text tokens in the first token sequence according to the indicator parameters to obtain the second token sequence; The inference module is used to perform inference based on the second token sequence to obtain the inference result of the multimodal data.

11. The apparatus according to claim 10, characterized in that, The second acquisition module is used to acquire a query Q matrix generated based on the tail token and a key K matrix generated based on the non-text token, wherein the Q matrix represents the information of interest in the inference process and the K matrix represents the features of the corresponding token; Based on the similarity between the Q matrix and the K matrix, the indicator parameters of the non-text token are obtained.

12. The apparatus according to claim 11, characterized in that, The second acquisition module is used to use the similarity between the Q matrix and the K matrix as an indicator parameter of the non-text token; or, to perform pooling or smoothing processing on the similarity between the Q matrix and the K matrix, and use the processed similarity between the Q matrix and the K matrix as an indicator parameter of any non-text token.

13. The apparatus according to claim 11 or 12, characterized in that, The Q matrix is ​​a compressed Q matrix. The number of attention heads in the uncompressed Q matrix is ​​greater than the number of attention heads in the K matrix. The number of attention heads in the compressed Q matrix is ​​the same as the number of attention heads in the K matrix.

14. The apparatus according to any one of claims 10-13, characterized in that, The first acquisition module is used to input the multimodal data into a multimodal large model, the multimodal large model including N layers of Transformers, where N is a positive integer; and to acquire the first token sequence based on the output of the (m-1)th layer of Transformers, where m is a positive integer less than or equal to N. The inference module is used to input the second token sequence into the m-th layer Transformer of the multimodal large model, and obtain the inference result of the multimodal data based on the output result of the multimodal large model.

15. The apparatus according to claim 14, characterized in that, The model variables in the Transformer at layer (m-1) and before layer (m-1) match the size of the first token sequence, and the model variables in the Transformer at layer (m) and after layer (m) match the size of the second token sequence.

16. The apparatus according to claim 14 or 15, characterized in that, The compression module is used to compress the non-text tokens in the first token sequence according to the index parameters and the compression ratio to obtain the second token sequence. The compression ratio is the ratio of the total number of non-text tokens in the second token sequence to the total number of non-text tokens in the first token sequence.

17. The apparatus according to claim 16, characterized in that, The values ​​of m and the compression ratio are the values ​​of the multimodal large model under the condition that the search conditions are met. The search conditions are: the inference accuracy of the multimodal large model is greater than the accuracy threshold, and the inference time of the multimodal large model is less than the time threshold.

18. The apparatus according to any one of claims 10-17, characterized in that, The device further includes: The identification module is used to identify non-text tokens in the first token sequence based on the mask sequence corresponding to the first token sequence. The length of the mask sequence is the same as the length of the first token sequence, and the mask sequence indicates the position of the non-text token in the first token sequence.

19. A multimodal data inference device, characterized in that, The multimodal data inference device includes a processor coupled to a memory, the memory storing at least one program instruction or code, the at least one program instruction or code being loaded and executed by the processor to enable the multimodal data inference device to implement the multimodal data inference method according to any one of claims 1-9.

20. A computer-readable storage medium, characterized in that, The computer storage medium stores at least one instruction, which is loaded and executed by a processor to enable the computer to implement the inference method for multimodal data as described in any one of claims 1-9.

21. A computer program product, characterized in that, The computer program product includes: computer program code, which is loaded and executed by a computer to enable the computer to implement the inference method for multimodal data as described in any one of claims 1-9.

22. A chip, characterized in that, The chip includes a processor for calling and executing instructions stored in a memory to enable a device equipped with the chip to implement the inference method for multimodal data as described in any one of claims 1-9.

Images