Multimedia data-based task execution methods, related devices, equipment, and media

Abstract

The present application discloses a method for executing a task based on multimedia data and related devices, equipment and media, wherein the method includes: obtaining fusion features based on the encoding features of the target multimedia data and the embedded features of the prompt text; performing attention calculation based on the output features of the previous decoding process to obtain the first hidden layer feature, and based on the feature vector of the target word in the first hidden layer feature, selecting the target feedforward layer from several candidate feedforward layers in the current decoding process and determining its weight factor, weighting the second hidden layer feature after the first hidden layer feature is processed by the target feedforward layer based on the weight factor of the target feedforward layer to obtain the third hidden layer feature, obtaining the output feature of the current decoding process based on the first hidden layer feature and the third hidden layer feature; obtaining the execution result based on the decoding results of each round of autoregressive decoding. The above scheme can reduce the computing resources required to execute multimedia tasks and improve the execution efficiency of multimedia tasks.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a task execution method and related apparatus, devices and media based on multimedia data. Background Technology

[0002] With the rapid development of artificial intelligence technology, multimodal large models are gradually being widely used in many scenarios such as education and healthcare.

[0003] Currently, in order to make multimodal large models applicable to complex and diverse multimedia tasks, the model parameters of multimodal large models are growing exponentially, leading to a corresponding sharp increase in the demand for computing resources, which in turn affects the execution efficiency of multimedia tasks. Therefore, how to reduce the computing resources required to execute multimedia tasks while improving their execution efficiency, while meeting the execution requirements of different multimedia tasks as much as possible, has become an urgent problem to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a task execution method and related apparatus, equipment, and medium based on multimedia data, which can reduce the computing resources required to execute multimedia tasks and improve the execution efficiency of multimedia tasks while meeting the execution requirements of different multimedia tasks as much as possible.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a task execution method based on multimedia data, comprising: obtaining fused features based on the encoding features of the target multimedia data and the embedding features of the prompt text; wherein, the prompt text describes the target task to be performed on the target multimedia data in natural language, and the decoding result of the fused features after multiple rounds of autoregressive decoding is used as the execution result of performing the target task on the target multimedia data, each round of autoregressive decoding comprising multiple decoding processes executed sequentially; during each round of autoregressive decoding: attention is calculated based on the output features of the previous decoding process of the current decoding process to obtain attention. The first hidden layer features are calculated, and based on the feature vector of the target word in the first hidden layer features, the target feedforward layer is selected from several candidate feedforward layers in the current decoding process, and the weight factor of the target feedforward layer is determined. Based on the weight factor of the target feedforward layer, the second hidden layer features after the first hidden layer features are weighted and processed by the target feedforward layer to obtain the third hidden layer features. Based on the first hidden layer features and the third hidden layer features, the output features of the current decoding process are obtained. The decoding result of this round of autoregressive decoding is predicted by the output features of the last decoding process in this round of autoregressive decoding. Based on the decoding results of each round of autoregressive decoding, the execution result is obtained.

[0006] To address the aforementioned technical problems, a second aspect of this application provides a task execution device based on multimedia data, comprising: a fusion module, a decoding module, and an acquisition module. The fusion module is used to obtain fusion features based on the encoding features of the target multimedia data and the embedding features of the prompt text; wherein the prompt text describes the target task to be performed on the target multimedia data in natural language, and the decoding result of the fusion features after multiple rounds of autoregressive decoding serves as the execution result of performing the target task on the target multimedia data. Each round of autoregressive decoding includes multiple sequentially executed decoding processes. The decoding module is used, during each round of autoregressive decoding, to: perform... Attention calculation is performed to obtain the first hidden layer features. Based on the feature vector of the target word in the first hidden layer features, a target feedforward layer is selected from several candidate feedforward layers in the current decoding process, and the weight factor of the target feedforward layer is determined. Based on the weight factor of the target feedforward layer, the second hidden layer features after the first hidden layer features are weighted and processed by the target feedforward layer to obtain the third hidden layer features. Based on the first hidden layer features and the third hidden layer features, the output features of the current decoding process are obtained. The decoding result of this round of autoregressive decoding is predicted by the output features of the last decoding process in this round of autoregressive decoding. The acquisition module is used to obtain the execution result based on the decoding results of each round of autoregressive decoding.

[0007] To address the aforementioned technical problems, a third aspect of this application provides an electronic device comprising at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the multimedia data-based task execution method described in the first aspect.

[0008] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the multimedia data-based task execution method of the first aspect described above.

[0009] The above scheme, based on the encoding features of the target multimedia data and the embedding features of the prompt text, obtains fused features. The prompt text describes the target task to be performed on the target multimedia data in natural language. The decoding result of the fused features after multiple rounds of autoregressive decoding is used as the execution result of the target task on the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially. In each round of autoregressive decoding: attention is calculated based on the output features of the previous decoding process to obtain the first hidden layer features for attention calculation. Based on the feature vector of the target word in the first hidden layer features, the target feedforward layer is selected from several candidate feedforward layers in the current decoding process, and the weight factor of the target feedforward layer is determined. Based on the weight factor of the target feedforward layer, the second hidden layer features after the first hidden layer features are weighted and processed by the target feedforward layer to obtain the third hidden layer features. Based on the first hidden layer features and the third hidden layer features, the output features of the current decoding process are obtained. The decoding result of this round of autoregressive decoding is predicted by the output features of the last decoding process in this round of autoregressive decoding. Therefore, based on the decoding results of each round of autoregressive decoding, the execution result is obtained. Thus, on the one hand, because in In the autoregressive decoding process, the target feedforward layer is adaptively selected from several candidate feedforward layers for feature processing. Compared to feature processing through all candidate feedforward layers, this approach controls the computational resources required while expanding model parameters to meet the execution needs of different multimedia tasks as much as possible. Furthermore, the selection of the target feedforward layer considers the feature information of both the target multimedia data and the prompt text. This allows for adaptive selection of the target feedforward layer even in scenarios where the prompt text is the same but the target multimedia data is different, or vice versa. Compared to selecting the target feedforward layer by only considering the feature information of either the target multimedia data or the prompt text, this approach further meets the execution needs of different multimedia tasks. Moreover, after selecting the target feedforward layer based on the feature vector of the target word and determining its weight factor, the selected target feedforward layer and its determined weight factor are shared among all words in the first hidden layer features. Compared to selecting the target feedforward layer and determining its weight factor separately for each word in the first hidden layer features, this approach significantly reduces the routing overhead during inference and helps improve the execution efficiency of multimedia tasks. Therefore, while meeting the execution requirements of different multimedia tasks as much as possible, it can reduce the computing resources required to execute multimedia tasks and improve the execution efficiency of multimedia tasks. Attached Figure Description

[0010] Figure 1 This is a flowchart illustrating an embodiment of the task execution method based on multimedia data in this application;

[0011] Figure 2 This is a schematic diagram of the framework of an embodiment of the multimodal large model of this application;

[0012] Figure 3 This is a schematic diagram of an embodiment of the feature replacement operation of this application;

[0013] Figure 4 This is a schematic diagram of an embodiment of the feature rearrangement operation of this application;

[0014] Figure 5 This is a schematic diagram of the framework of an embodiment of the task execution device based on multimedia data according to this application;

[0015] Figure 6 This is a schematic diagram of the framework of an embodiment of the electronic device of this application;

[0016] Figure 7 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0017] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0018] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0019] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the slash " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper indicates two or more objects.

[0020] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the task execution method based on multimedia data according to this application. Specifically, it may include the following steps:

[0021] Step S11: Based on the encoding features of the target multimedia data and the embedding features of the prompt text, obtain the fusion features.

[0022] In this embodiment of the disclosure, the prompt text can describe the target task to be performed on the target multimedia data in natural language. For example, if the target task is complex, the prompt text can describe the target task using long sentences; or, if the target task is relatively simple, the prompt text can describe the target task using short sentences.

[0023] In an implementation scenario, the target multimedia data can include, but is not limited to, image data, audio data, video data, animation data, etc. The specific type of target multimedia data is not limited here. It should be noted that although the target multimedia data is named "multimedia data," it is not limited to containing two or more different types of data; it can also contain a single type of data. For example, target multimedia data can contain only image data, or it can contain only audio data. Of course, target multimedia data can also contain two or more different types of data. For example, target multimedia data can be video data, in which case it includes audio and image data, and may even contain text data (i.e., video subtitles). The above examples are merely a few possible examples of target multimedia data; the specific type of target multimedia data is not limited here, nor will all examples be provided.

[0024] In an implementation scenario, the target task can include, but is not limited to, image recognition, text recognition, audio recognition, video analysis, and image expansion tasks, etc. The specific type of target task is not limited here. Taking the target multimedia data as image data and the target task as image recognition as an example, the prompt text can include, but is not limited to, the following: "Please identify and output whether there is a target XX in the image and, if so, the specific number of targets XX" (i.e., in this case, the image recognition task specifically identifies targets in the image), etc. Of course, the above examples are merely possible examples of the specific content of the prompt text when using image recognition as an example in practical applications. Other possible situations are not limited here, nor will they be listed one by one.

[0025] In one implementation scenario, the encoded features of the target multimedia data can be obtained by feature encoding by an encoder. For example, the encoder can include, but is not limited to, convolutional neural networks, variational autoencoders, convolutional autoencoders, visual Transformer encoders, or cross-modal encoders such as CLIP (Contrastive Language-Image Pretraining). The network type of the encoder is not limited here, nor will it be listed in detail.

[0026] In one implementation scenario, the embedded features of the prompt text can be obtained by feature embedding in an embedding layer. For example, feature embedding can be achieved through methods including but not limited to: statistical encoding, word vector models, pre-trained language models, etc. The implementation methods of feature embedding are not limited here, nor will they be listed one by one.

[0027] In one implementation scenario, as a possible example, after obtaining the encoded features and embedded features, a weighted average can be performed on the encoded features and embedded features to obtain the fused features. Alternatively, as another possible example, after obtaining the encoded features and embedded features, a cross-attention mechanism can be used to process the encoded features and embedded features to achieve feature interaction between the encoded features and embedded features. The output feature processed by the cross-attention mechanism can then be used as the fused feature. It should be noted that the fused feature can contain feature information from both the target multimedia data and the prompt text. Furthermore, the above examples are merely a few possible examples of obtaining the fused feature of encoded features and embedded features; other possible methods for obtaining this feature in practical applications are not limited here. For example, in practical applications, the fusion method of encoded features and embedded features should preferably be such that the fused feature can contain feature information from both the target multimedia data and the prompt text.

[0028] It should be noted that the decoding results of the fused features after multiple rounds of autoregressive decoding in this embodiment can be used as the execution result of the target task on the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially. The core idea of ​​autoregressive decoding technology is to construct a complete sequence through "word-by-word generation and circular dependency," that is, the decoding result of each round of autoregressive decoding depends on the decoding results of the previous rounds of autoregressive decoding, until the termination condition is met (e.g., the decoding result of a certain round of autoregressive decoding is an end symbol, or the accumulated decoding results up to this round of autoregressive decoding have reached the maximum length, etc.). In this embodiment, in order to avoid redundancy in the scheme description, the specific method of "word-by-word generation and circular dependency" in the autoregressive decoding technology is described in detail. In practical applications, each round of autoregressive decoding can obtain the decoding results of the previous rounds of autoregressive decoding and apply these decoding results to the current round of autoregressive decoding. For example, these decoding results can be used to extract feature vectors and fused together with the encoding features of the target multimedia data and the embedding features of the prompt text to obtain fused features, which are used as the input features of the current round of autoregressive decoding. The specific process can be referred to the technical details of autoregressive decoding technology, which will not be repeated here. For better understanding, please refer to the following: Figure 2 , Figure 2 This is a schematic diagram of the framework of an embodiment of the multimodal large model of this application. For example... Figure 2As shown, a multimodal large model can include an encoder, an embedding layer (not shown), and a decoder. The encoder is used to encode the target multimedia data, and the embedding layer is used to embed the prompt text. It should be noted that the encoder and embedding layer can be referred to the aforementioned descriptions. Furthermore, the decoder can be used to perform autoregressive decoding. For example, the decoder can include, but is not limited to, L-layer stacked Transformer modules, etc., and the network structure of the decoder is not limited here. Taking a decoder including L-layer stacked Transformer modules as an example, each round of autoregressive decoding requires passing through L-layer stacked Transformer modules, and each decoding process in each round of autoregressive decoding corresponds to the processing flow of different layers of Transformer modules (e.g., ...). Figure 2 (As shown in the rectangular block filled with diagonal shading). Exemplarily, in this embodiment of the disclosure, the Transformer module may include an attention layer and a feedforward layer. In practical applications, the Transformer module may also include other network layers, such as a normalization layer, etc., and is not limited herein.

[0029] Step S12: During each round of autoregressive decoding: attention is calculated based on the output features of the previous decoding process in the current decoding process to obtain the first hidden layer features of attention calculation. Based on the feature vector of the target word in the first hidden layer features, the target feedforward layer is selected from several candidate feedforward layers in the current decoding process and the weight factor of the target feedforward layer is determined. Based on the weight factor of the target feedforward layer, the second hidden layer features after the first hidden layer features are processed by the target feedforward layer are weighted to obtain the third hidden layer features. Based on the first hidden layer features and the third hidden layer features, the output features of the current decoding process are obtained.

[0030] In this embodiment of the disclosure, the decoding result of the current round of autoregressive decoding is predicted by the output features of the last decoding process in the current round of autoregressive decoding. For example, the output features of the last decoding process in the current round of autoregressive decoding can be predicted through network layers such as linear layers and softmax layers to obtain the decoding result of the current round of autoregressive decoding. Taking a decoder consisting of L stacked Transformer modules as an example, the last Transformer module in the decoder can be followed by a linear layer and a softmax layer. The output features of the last Transformer module can then be processed sequentially through the subsequent linear layer and softmax layer to predict the decoding result of the current round of autoregressive decoding. For details, please refer to the technical details of Transformer-based decoders, which will not be elaborated here.

[0031] In one implementation scenario, for each decoding process in each round of autoregressive decoding, attention can be calculated first based on the output features of the previous decoding process to obtain the first hidden layer features for attention calculation. It should be noted that when the current decoding process is the first decoding process in this round of autoregressive decoding, since there is no previous decoding process, the fused features can be directly used as the input features of the current decoding process; that is, attention can be calculated directly based on the fused features to obtain the first hidden layer features for attention calculation. Furthermore, attention calculation can be implemented through attention mechanisms including, but not limited to, self-attention mechanisms and multi-head self-attention mechanisms. Specific details can be found in the technical details of the aforementioned attention mechanisms, which will not be elaborated here. If this round of autoregressive decoding is not the first round, feature vectors can be extracted based on the decoding results obtained in previous rounds of autoregressive decoding, and fused together with the encoded features and embedded features to obtain fused features, which can then be used as the input features for this round of autoregressive decoding. Specific details can be found in the aforementioned descriptions, which will not be repeated here.

[0032] In one implementation scenario, after obtaining the first hidden layer features, the target feedforward layer can be selected from several candidate feedforward layers in the current decoding process based on the feature vectors of the target words in the first hidden layer features, and the weight factor of the target feedforward layer can be determined. It should be noted that different candidate feedforward layers can be applied to different tasks and different data patterns. For example, at least one candidate feedforward layer can be applied to audio data recognition tasks, at least one candidate feedforward layer can be applied to image data recognition tasks, at least one candidate feedforward layer can be applied to audio data translation tasks, and at least one candidate feedforward layer can be applied to video data analysis tasks; these will not be listed individually here. Furthermore, the network structure of the candidate feedforward layer can be the same as (or different from) the standard feedforward layer. The main difference between them lies in the network parameters trained on a specific training set, making them suitable for the application scenario represented by that specific training set. For example, a standard feedforward layer can be trained based on sample image data for an image recognition task to obtain a candidate feedforward layer, which can then be applied to image data recognition tasks. Candidate feedforward layers suitable for other application scenarios can be trained in the same way, and will not be elaborated further here.

[0033] In a specific implementation scenario, the target word can be the last word in the first hidden layer features. For ease of understanding, let's take an example where the first hidden layer features contain *s* words, each represented by a *d*-dimensional feature vector. The target word is the *s*-th word in the first hidden layer features, and its feature vector is the *d*-dimensional feature vector of the *s*-th word in the first hidden layer features. This approach, by setting the target word as the last word in the first hidden layer features, allows for reviewing historical information to aid autoregressive decoding.

[0034] In a specific implementation scenario, after obtaining the feature vector of the target word in the first hidden layer features, prediction can be performed based on the feature vector of the target word to obtain the selection scores of several candidate feedforward layers. For example, the feature vector of the target word can be predicted using a fully connected layer to obtain the selection scores of several candidate feedforward layers. It should be noted that the selection scores of the candidate feedforward layers can characterize the suitability of the candidate feedforward layers in processing the first hidden layer features. For example, the higher the selection score of the candidate feedforward layer, the more suitable it is in processing the first hidden layer features; conversely, the lower the selection score, the less suitable it is. Based on this, the target feedforward layer can be selected from among the candidate feedforward layers and its weight factor can be determined. Specifically, several candidate feedforward layers can be sorted based on their selection scores, and the top-ranked (e.g., top 2, top 3, etc.) candidate feedforward layers can be selected as the target feedforward layers. Then, the weight factor of the target feedforward layer is determined based on its selection score, and candidate feedforward layers not selected as target feedforward layers are configured with a weight factor of 0. For example, the candidate feedforward layers can be sorted in descending order of selection scores, and the candidate feedforward layers ranked before a predetermined position (e.g., top 2, top 3, etc.) can be selected as the target feedforward layers. Based on this, the selection score of the target feedforward layer can be used as its weight factor, or the selection scores of each target feedforward layer can be normalized to obtain the target feedforward layer's weight factor. Of course, the above examples are merely a few possible ways to obtain the weight factor; the method of measuring the weight factor is not limited here, nor will it be listed in detail. For ease of understanding, taking the l-th decoding process as an example, the feature vector of the target word (i.e., the s-th word) in the first hidden layer features is denoted as... For example, the selection score s of the i-th candidate feedforward layer i It can be represented as:

[0035]

[0036] In the above formula, f c Let represent a fully connected layer and ... i It can be represented as:

[0037]

[0038] In the above formula, Topk represents the selection score s1 from the first candidate feedforward layer to the selection score snn from the Nth candidate feedforward layer. NThe K candidate feedforward layers with the highest scores are selected as the target feedforward layers, and their weight factors are configured according to their selection scores. That is, in this case, the weight factor g of the i-th target feedforward layer is... i That is, to select a score s for it. i Furthermore, besides the aforementioned K candidate feedforward layers, the weight factors for other candidate feedforward layers are configured to 0. This method predicts based on the feature vectors of the target words, obtaining selection scores for each of the candidate feedforward layers. These selection scores represent the suitability of each candidate feedforward layer in processing the first hidden layer features. Based on these selection scores, the target feedforward layer is selected from among the candidate feedforward layers, and its weight factors are determined. This approach maximizes the adaptability of the target feedforward layer to the target multimedia data and its target task.

[0039] In one implementation scenario, after determining the target feedforward layer and its weight factors, the feature vectors of the current word in the first hidden layer features, processed by the corresponding target feedforward layer, are weighted and summed based on the weight factors of each target feedforward layer to obtain the weighted vector of the current word. On this basis, the weighted vectors of each word in the first hidden layer features can be retrieved from contiguous memory and combined sequentially to obtain the third hidden layer features. This method, where each word in the first hidden layer features shares the target feedforward layer and its weight factors, saves routing overhead as much as possible compared to selecting a target feedforward layer and determining weight factors for each word separately, thus helping to improve the inference efficiency of task execution.

[0040] In a specific implementation scenario, each word in the first hidden layer features can be treated as the current word. The feature vectors of these current words are then processed by each target feedforward layer. After processing, the feature vectors processed by each target feedforward layer are weighted and summed using their respective weight factors to obtain the weighted vector of the current word. Finally, the weighted vectors of each word in the first hidden layer features are retrieved from contiguous memory and combined sequentially to obtain the third hidden layer features. For example, the weighted vectors of the 1st word, the 2nd word, ..., the sth word are combined sequentially to obtain the third hidden layer features.

[0041] In a specific implementation scenario, for ease of description, please refer to the relevant documentation. Figure 2 Taking the l-th decoding process as an example, for the t-th word in the first hidden layer features, its feature vector can be denoted as: Then its weighted vector can be expressed as: Among them, g i Representing each candidate feedforward layer (e.g.) Figure 2The weight factors of the rectangle filled with dotted shadows (as mentioned earlier) are set to 0 for candidate feedforward layers that are not selected as the target feedforward layer. This means that these candidate feedforward layers are not being used to process word vectors. (FNN) i Let N represent the i-th candidate feedforward layer, and N represent the total number of candidate feedforward layers.

[0042] In one implementation scenario, as a possible approach, after obtaining the first and third hidden layer features, the features can be fused together to obtain the output features of the current decoding process. For example, the first and third hidden layer features can be summed to obtain the output features of the current decoding process. Alternatively, as another possible implementation, please refer to [further details omitted]. Figure 2 The current decoding process can also be configured with several shared feedforward layers (such as...). Figure 2 The rectangle filled with a grid shaded area can be used to process the first hidden layer features based on several shared feedforward layers in the current decoding process. This yields the fourth hidden layer features after each shared feedforward layer has processed the first hidden layer features. These fourth hidden layer features are then summed to obtain the fifth hidden layer features. Based on this, the first, third, and fifth hidden layer features can be fused to obtain the output features of the current decoding process. For example, the summation of the first, third, and fifth hidden layer features can be used as the output features of the current decoding process. For clarity, please refer to the following documentation. Figure 2 Taking the l-th decoding process as an example, for the t-th word in the first hidden layer features, its feature vector can be denoted as: Then, its feature vector, after being processed by each shared feedforward layer and then accumulated, can be expressed as: Among them, FNN i K represents the i-th shared feedforward layer. s This represents the total number of shared feedforward layers. Furthermore, for the t-th word in the first hidden layer features, its output vector after passing through the candidate feedforward layer and the shared feedforward layer is... It can be represented as:

[0043]

[0044] As can be seen from the above formula, the feature vector of the t-th word in the output features of the current decoding process is the aforementioned output feature. In other words, the output features of each word in the first hidden layer after the above processing can be combined to obtain the output features of the current decoding process. Similar to the candidate feedforward layer, the network structure of the shared feedforward layer can also refer to the standard feedforward layer, which will not be elaborated here. In addition, unlike the candidate feedforward layer, all inputs "globally share" the feedforward layer without routing part of it. That is, the shared feedforward layer does not rely on the routing mechanism. It is mainly responsible for capturing common and general feature information in the input features (such as basic grammar rules, low-level visual features, etc.). For details, please refer to the technical details of the shared expert, which will not be elaborated here.

[0045] It should be noted that the above implementation process is the specific flow of the output features of the current decoding process. In practical applications, each decoding process in each round of autoregressive decoding can have a candidate feedforward layer, and even a shared feedforward layer can be further set; or, in each round of autoregressive decoding, at least one decoding process can be selected at a target interval to have a candidate feedforward layer. For example, a candidate feedforward layer can be set in every other decoding process, and a shared feedforward layer can be set at the same time as the candidate feedforward layer. Of course, only a candidate feedforward layer can be set, which is not limited here. To address the above situation, after obtaining the first hidden layer features and before selecting the target feedforward layer and determining its weight factor, it is possible to first detect whether there are several candidate feedforward layers or only one feedforward layer in the current decoding process. In response to the current decoding process having several candidate feedforward layers, the steps of selecting the target feedforward layer and determining the weight factor of the target feedforward layer based on the feature vector of the target word in the first hidden layer features can be performed. For details, please refer to the aforementioned relevant descriptions, which will not be repeated here. Conversely, in response to the current decoding process having only one feedforward layer, the first hidden layer features and the sixth hidden layer features after the first hidden layer features have been processed by the feedforward layer in the current decoding process can be fused (e.g., added) to obtain the output features of the current decoding process.

[0046] To facilitate understanding of the routing overhead saving effect of the decoding process described in the embodiments of this disclosure, another related implementation method is introduced below as a comparative example of the decoding process described in the embodiments of this disclosure. Unlike the decoding process in the embodiments of this disclosure, in the current decoding process of the comparative example, for the feature vector of each word in the first hidden layer features, the following process steps can be repeatedly executed: Based on the feature vector of the current word, prediction is performed to obtain selection scores for several candidate feedforward layers; based on the selection scores of several candidate feedforward layers, a target feedforward layer is selected from the several candidate feedforward layers, and the weight factor of the target feedforward layer is determined; based on the weight factor of the target feedforward layer, the feature vector after processing by the target feedforward layer is weighted and summed to obtain the weighted vector of the current word; and based on the weighted vector of the current word and the feature vector of the current word, they are fused (e.g., added) to obtain the output vector of the current word in the current decoding process, and the output features of the current decoding process are obtained by combining the output vectors of each word. To facilitate understanding and comparison of the example with the decoding process described in the embodiments of this disclosure, the feature vector of the t-th word in the first hidden layer features during the l-th decoding process will still be used. For example, the selection score s of the i-th candidate feedforward layer i,t It can be represented as:

[0047]

[0048] Based on this, the weight factors of each candidate feedforward layer can be determined, where the weight factor g of the i-th candidate feedforward layer is determined when the t-th word is processed in the decoding process. i,t It can be represented as:

[0049]

[0050] Based on this, the output vector of the t-th word in the first hidden layer features during the l-th decoding process It can be represented as:

[0051]

[0052] Of course, if a shared feedforward layer is also included in the l-th decoding process, the output vector of the t-th word in the first hidden layer features during the l-th decoding process... It can be represented as:

[0053]

[0054] Therefore, the main difference between the decoding process in the comparative example and the embodiments of this disclosure is that in the comparative example, the target feedforward layer and its weight factors are not shared across all words in the first hidden layer features; instead, each word determines its own target feedforward layer and its weight factors. Please continue to refer to the following... Figure 3 and Figure 4 , Figure 3 This is a schematic diagram of an embodiment of the feature replacement operation of this application. Figure 4 This is a schematic diagram of an embodiment of the feature rearrangement operation of this application. For example... Figure 3 As shown, taking a first hidden layer feature containing 5 words (denoted as word 0, word 1, word 2, word 3, and word 4 respectively) and a total of 4 candidate feedforward layers (denoted as candidate feedforward layer 0, candidate feedforward layer 1, candidate feedforward layer 2, and candidate feedforward layer 3 respectively) as an example, their respective feature vectors are illustrated by rectangles filled with different shades. If the top two scores are selected as the target feedforward layers, then word 0 is predicted to determine candidate feedforward layer 1 and candidate feedforward layer 3 as the target feedforward layers for word 0; word 1 is predicted to determine candidate feedforward layer 0 and candidate feedforward layer 3 as the target feedforward layers for word 1; word 2 is predicted to determine candidate feedforward layer 0 and candidate feedforward layer 1 as the target feedforward layers for word 2; word 3 is predicted to determine candidate feedforward layer 1 and candidate feedforward layer 3 as the target feedforward layers for word 3; and word 4 is predicted to determine candidate feedforward layer 1 and candidate feedforward layer 1 as the target feedforward layers for word 4. Based on this, the feature vectors to be processed by the same feedforward layer are converged. Specifically, candidate feedforward layer 0 converges the feature vectors of word 1, word 2, and word 4 for processing; candidate feedforward layer 1 converges the feature vectors of word 0, word 2, word 3, and word 4 for processing; candidate feedforward layer 3 converges the feature vectors of word 0, word 1, and word 3 for processing; while candidate feedforward layer 2 has no feature vectors to be processed. After the calculation is completed, the following steps need to be performed: Figure 4 The rearrangement operation shown feeds back the computation results of each candidate feedforward layer to the original sample and reassembles them according to the original order of each word. For example... Figure 4 In the process, after the feature vectors of word 1, word 2, and word 4 are processed separately in candidate feedforward layer 0, they can be further processed using their respective weight factors (e.g., ...). Figure 4 The feature vectors of lexical units 1, 2, and 4, marked with numbers 0.2, 0.9, and 0.7 respectively, are multiplied by the processed feature vectors in candidate feedforward layer 0. Similarly, after candidate feedforward layer 1 processes the feature vectors of lexical units 0, 2, 3, and 4, they can be further multiplied by their respective weight factors (e.g., ...). Figure 4 The feature vectors of the four words (0.3, 0.1, 0.4, and 0.3) to be processed in the candidate feedforward layer 1 are multiplied by the processed feature vectors respectively. Similarly, after the candidate feedforward layer 3 processes the feature vectors of words 0, 1, and 3 respectively, it can continue to multiply them by their respective weight factors (e.g., ...). Figure 4The feature vectors of the three words (0.7, 0.8, and 0.6) that need to be processed by the candidate feedforward layer 3 are multiplied by the processed feature vectors respectively. On this basis, the feature vectors that have been weighted by the weighting factors are then recombined according to the original order of word 0, word 1, word 2, and word 3. For example, for word 0, its weighted feature vectors after being weighted by candidate feedforward layer 1 and candidate feedforward layer 3 can be recombined to obtain the weighted vector of word 0; similarly, for word 1, its weighted feature vectors after being weighted by candidate feedforward layer 0 and candidate feedforward layer 3 can be recombined to obtain the weighted vector of word 1; similarly, for word 2, its weighted feature vectors after being weighted by candidate feedforward layer 1 and candidate feedforward layer 3 can be recombined to obtain the weighted vector of word 2; similarly, for word 3, its weighted feature vectors after being weighted by candidate feedforward layer 0 and candidate feedforward layer 1 can be recombined to obtain the weighted vector of word 3. Figure 3 and Figure 4 As shown, in the feature permutation operation of the comparative example, the tokens need to be grouped according to the candidate feedforward layers, and data needs to be read non-contiguously from contiguous memory (e.g., tokens 1, 2, 4 → candidate feedforward layer 0). In the rearrangement operation of the comparative example, the calculation results of the candidate feedforward layers need to be reassembled in the original order. At this time, random writing to memory is required. However, the random access bandwidth of GPU memory is only 1 / 10 to 1 / 5 of that of continuous access. Due to this hardware limitation, compared with the decoding process in this embodiment, the comparative example will exacerbate the latency and cannot fully utilize network bandwidth, resulting in low execution efficiency of multimedia tasks. In other words, in the decoding process of this embodiment, since the target feedforward layer and its weight factors are shared among the tokens of the first hidden layer features, the overall cost of a single route can be reduced as the degree of autoregressive decoding increases. In addition, since the decoding process in this embodiment does not shuffle the token order, only the entire contiguous memory needs to be copied during the feature permutation operation, and no random writing to memory is required during the rearrangement operation, thus enabling continuous memory access. In particular, in the embodiments of this disclosure, when the batch number is 1, the feature permutation operation and the recombination operation can be omitted.

[0055] Step S13: Based on the decoding results of each round of autoregressive decoding, obtain the execution result.

[0056] Specifically, at the end of the autoregressive decoding, the decoding results of each round of autoregressive decoding can be combined sequentially to obtain the execution result. For example, when the target task is text recognition of image data, each round of autoregressive decoding yields a decoded character as the decoding result for that round. These decoded characters from each round can then be combined sequentially to obtain the execution result of the text recognition task (i.e., the text content in the image data). Similarly, when the target task is audio recognition of audio data, each round of autoregressive decoding yields a decoded character as the decoding result for that round. These decoded characters from each round can then be combined sequentially to obtain the execution result of the audio recognition task (i.e., the recognized text in the audio data). It should be noted that the above examples are merely a few possible examples of execution results when the target multimedia data is of different data types and the target tasks are different. Other possible scenarios are not limited here, nor will they be listed in detail.

[0057] The above scheme, based on the encoding features of the target multimedia data and the embedding features of the prompt text, obtains fused features. The prompt text describes the target task to be performed on the target multimedia data in natural language. The decoding result of the fused features after multiple rounds of autoregressive decoding is used as the execution result of the target task on the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially. In each round of autoregressive decoding: attention is calculated based on the output features of the previous decoding process to obtain the first hidden layer features for attention calculation. Based on the feature vector of the target word in the first hidden layer features, the target feedforward layer is selected from several candidate feedforward layers in the current decoding process, and the weight factor of the target feedforward layer is determined. Based on the weight factor of the target feedforward layer, the second hidden layer features after the first hidden layer features are weighted and processed by the target feedforward layer to obtain the third hidden layer features. Based on the first hidden layer features and the third hidden layer features, the output features of the current decoding process are obtained. The decoding result of this round of autoregressive decoding is predicted by the output features of the last decoding process in this round of autoregressive decoding. Therefore, based on the decoding results of each round of autoregressive decoding, the execution result is obtained. Thus, on the one hand, because in In the autoregressive decoding process, the target feedforward layer is adaptively selected from several candidate feedforward layers for feature processing. Compared to feature processing through all candidate feedforward layers, this approach controls the computational resources required while expanding model parameters to meet the execution needs of different multimedia tasks as much as possible. Furthermore, the selection of the target feedforward layer considers the feature information of both the target multimedia data and the prompt text. This allows for adaptive selection of the target feedforward layer even in scenarios where the prompt text is the same but the target multimedia data is different, or vice versa. Compared to selecting the target feedforward layer by only considering the feature information of either the target multimedia data or the prompt text, this approach further meets the execution needs of different multimedia tasks. Moreover, after selecting the target feedforward layer based on the feature vector of the target word and determining its weight factor, the selected target feedforward layer and its determined weight factor are shared among all words in the first hidden layer features. Compared to selecting the target feedforward layer and determining its weight factor separately for each word in the first hidden layer features, this approach significantly reduces the routing overhead during inference and helps improve the execution efficiency of multimedia tasks. Therefore, while meeting the execution requirements of different multimedia tasks as much as possible, it can reduce the computing resources required to execute multimedia tasks and improve the execution efficiency of multimedia tasks.

[0058] Please see Figure 5 , Figure 5This is a schematic diagram of a framework of an embodiment of the multimedia data-based task execution device of this application. The multimedia data-based task execution device 50 includes: a fusion module 51, a decoding module 52, and an acquisition module 53. The fusion module 51 is used to obtain fusion features based on the encoding features of the target multimedia data and the embedding features of the prompt text; wherein, the prompt text describes the target task to be performed on the target multimedia data in natural language, and the decoding result of the fusion features after multiple rounds of autoregressive decoding is used as the execution result of performing the target task on the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially. The decoding module 52 is used, during each round of autoregressive decoding, to perform attention calculation based on the output features of the previous decoding process of the current decoding process, to obtain... The first hidden layer feature is calculated based on the attention. Based on the feature vector of the target word in the first hidden layer feature, the target feedforward layer is selected from several candidate feedforward layers in the current decoding process, and the weight factor of the target feedforward layer is determined. Based on the weight factor of the target feedforward layer, the second hidden layer feature after the first hidden layer feature is processed by the target feedforward layer is weighted to obtain the third hidden layer feature. Based on the first hidden layer feature and the third hidden layer feature, the output feature of the current decoding process is obtained. The decoding result of this round of autoregressive decoding is predicted by the output feature of the last decoding process in this round of autoregressive decoding. The acquisition module 53 is used to obtain the execution result based on the decoding results of each round of autoregressive decoding.

[0059] The above scheme, based on the multimedia data task execution device 50, obtains fused features based on the encoding features of the target multimedia data and the embedding features of the prompt text. The prompt text describes the target task to be performed on the target multimedia data in natural language. The decoding result of the fused features after multiple rounds of autoregressive decoding is used as the execution result of the target task on the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially. In each round of autoregressive decoding: attention is calculated based on the output features of the previous decoding process to obtain the first hidden layer features for attention calculation. Based on the feature vector of the target word in the first hidden layer features, the target feedforward layer is selected from several candidate feedforward layers in the current decoding process, and the weight factor of the target feedforward layer is determined. Based on the weight factor of the target feedforward layer, the second hidden layer features after the first hidden layer features are weighted and processed by the target feedforward layer to obtain the third hidden layer features. Based on the first hidden layer features and the third hidden layer features, the output features of the current decoding process are obtained. The decoding result of this round of autoregressive decoding is predicted by the output features of the last decoding process in this round of autoregressive decoding. Finally, based on the decoding results of each round of autoregressive decoding, the execution result is obtained. Therefore, on the one hand, since the target feedforward layer is adaptively selected from several candidate feedforward layers for feature processing during the autoregressive decoding process, compared with feature processing through all candidate feedforward layers, it can control the required computing resources while expanding the model parameters to meet the execution requirements of different multimedia tasks as much as possible. On the other hand, the selection of the target feedforward layer takes into account the feature information of both the target multimedia data and the prompt text. Thus, the target feedforward layer can be adaptively selected in different scenarios where the prompt text is the same but the target multimedia data is different, or in different scenarios where the target multimedia data is the same but the prompt text is different. Compared with selecting the target feedforward layer by only referring to the feature information of either the target multimedia data or the prompt text, it can further meet the execution requirements of different multimedia tasks. Furthermore, after selecting the target feedforward layer based on the feature vector of the target word and determining the weight factor of the target feedforward layer, the selected target feedforward layer and its determined weight factor are shared among all words in the first hidden layer features. Compared with selecting the target feedforward layer and determining its weight factor for each word in the first hidden layer features separately, it can significantly reduce the routing overhead in the inference process and help improve the execution efficiency of multimedia tasks. Therefore, while meeting the execution requirements of different multimedia tasks as much as possible, it can reduce the computing resources required to execute multimedia tasks and improve the execution efficiency of multimedia tasks.

[0060] In some disclosed embodiments, the decoding module 52 includes a score prediction submodule, which is used to predict based on the feature vector of the target word to obtain the selection score of each of the several candidate feedforward layers; wherein, the selection score of the candidate feedforward layer represents the suitability of the candidate feedforward layer in processing the first hidden layer features; the decoding module 52 includes a selection determination submodule, which is used to select the target feedforward layer from the several candidate feedforward layers and determine the weight factor of the target feedforward layer based on the selection scores of each of the several candidate feedforward layers.

[0061] In some disclosed embodiments, the selection determination submodule includes a sorting selection unit, used to sort several candidate feedforward layers based on their respective selection scores, and select the candidate feedforward layer with a preset high position as the target feedforward layer; the selection determination submodule includes a weight determination unit, used to determine the weight factor of the target feedforward layer based on the selection score of the target feedforward layer; wherein, candidate feedforward layers that are not selected as the target feedforward layer are configured with a weight factor of 0.

[0062] In some disclosed embodiments, the decoding module 52 includes a weighted summation submodule, which is used to perform weighted summation on the feature vector of the current word in the first hidden layer feature after processing by the corresponding target feedforward layer based on the weight factors of each target feedforward layer, so as to obtain the weighted vector of the current word; the decoding module 52 includes a memory reading submodule, which is used to retrieve the weighted vectors of each word in the first hidden layer feature from the contiguous memory and combine them in sequence to obtain the third hidden layer feature.

[0063] In some disclosed embodiments, the decoding module 52 includes a shared feedforward submodule, used to process the first hidden layer features based on several shared feedforward layers in the current decoding process, to obtain the fourth hidden layer features after the first hidden layer features have been processed by each shared feedforward layer; the decoding module 52 includes a feature accumulation submodule, used to accumulate the fourth hidden layer features after processing by each shared feedforward layer, to obtain the fifth hidden layer features; the decoding module 52 includes a hidden layer fusion submodule, used to fuse the first hidden layer features, the third hidden layer features and the fifth hidden layer features to obtain the output features of the current decoding process.

[0064] In some disclosed embodiments, the decoding module 52 includes a feedforward detection submodule for detecting whether there are several candidate feedforward layers or only one feedforward layer in the current decoding process; the decoding module 52 includes a first response submodule for responding to the presence of several candidate feedforward layers in the current decoding process, performing the steps of selecting a target feedforward layer from several candidate feedforward layers in the current decoding process and determining the weight factor of the target feedforward layer based on the feature vector of the target word in the first hidden layer features.

[0065] In some disclosed embodiments, the decoding module 52 includes a second response submodule, which, in response to the fact that there is only one feedforward layer in the current decoding process, fuses the first hidden layer features and the sixth hidden layer features after the first hidden layer features have been processed by the feedforward layer in the current decoding process to obtain the output features of the current decoding process.

[0066] In some disclosed embodiments, at least one decoding process is selected from a plurality of decoding processes at a target interval, and a plurality of candidate feedforward layers are provided.

[0067] In some disclosed embodiments, the target multimedia data includes at least one of image data, audio data, video data, and animation data; and / or, the target task includes at least one of image recognition task, text recognition task, audio recognition task, video analysis task, and screen expansion task; and / or, the target lexical unit is the last lexical unit in the first hidden layer features; and / or, the execution result is obtained by a multimodal large model based on the target multimedia data and the prompt text. The multimodal large model includes an encoder, an embedding layer, and a decoder. The encoder is used to perform feature encoding on the target multimedia data, the embedding layer is used to embed text into the prompt text, and the decoder is used to perform autoregressive decoding.

[0068] Please see Figure 6 , Figure 6 This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 60 includes at least a memory 61 and a processor 62 coupled to each other. The memory 61 stores at least program instructions, and the processor 62 is used to execute the program instructions to implement the steps in any of the above-described embodiments of the task execution method based on multimedia data. For details, please refer to the foregoing disclosed embodiments, which will not be repeated here. The electronic device 60 may include, but is not limited to, mobile phones, tablet computers, learning machines, translation machines, smart screens, servers, etc., and the specific type of the electronic device 60 is not limited here.

[0069] Specifically, processor 62 controls itself and memory 61 to implement the steps in any of the above-described embodiments of the multimedia data-based task execution method. Processor 62 may also be referred to as a CPU (Central Processing Unit). Processor 62 may be an integrated circuit chip with signal processing capabilities. Processor 62 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or any conventional processor. Furthermore, processor 62 may be implemented using integrated circuit chips.

[0070] In the above scheme, the electronic device 60 obtains fused features based on the encoding features of the target multimedia data and the embedding features of the prompt text. The prompt text describes the target task to be performed on the target multimedia data in natural language. The decoding result of the fused features after multiple rounds of autoregressive decoding is used as the execution result of the target task on the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially. In each round of autoregressive decoding: attention is calculated based on the output features of the previous decoding process to obtain the first hidden layer features of attention calculation. Based on the feature vector of the target word in the first hidden layer features, the target feedforward layer is selected from several candidate feedforward layers in the current decoding process, and the weight factor of the target feedforward layer is determined. Based on the weight factor of the target feedforward layer, the second hidden layer features after the first hidden layer features are weighted by the second hidden layer features after the target feedforward layer processing to obtain the third hidden layer features. Based on the first hidden layer features and the third hidden layer features, the output features of the current decoding process are obtained. The decoding result of this round of autoregressive decoding is predicted by the output features of the last decoding process in this round of autoregressive decoding. Therefore, based on the decoding results of each round of autoregressive decoding, the execution result is obtained. Because the target feedforward layer is adaptively selected from several candidate feedforward layers for feature processing during autoregressive decoding, compared to feature processing through all candidate feedforward layers, it can control the required computational resources while expanding model parameters to meet the execution requirements of different multimedia tasks as much as possible. On the other hand, the selection of the target feedforward layer takes into account the feature information of both the target multimedia data and the prompt text. Thus, the target feedforward layer can be adaptively selected in different scenarios where the prompt text is the same but the target multimedia data is different, or in different scenarios where the target multimedia data is the same but the prompt text is different. Compared to selecting the target feedforward layer by only referring to the feature information of either the target multimedia data or the prompt text, it can further meet the execution requirements of different multimedia tasks. Furthermore, after selecting the target feedforward layer based on the feature vector of the target word and determining the weight factor of the target feedforward layer, the selected target feedforward layer and its determined weight factor are shared among all words in the first hidden layer features. Compared to selecting the target feedforward layer and determining its weight factor separately for each word in the first hidden layer features, it can significantly reduce the routing overhead in the inference process and help improve the execution efficiency of multimedia tasks. Therefore, while meeting the execution requirements of different multimedia tasks as much as possible, it can reduce the computing resources required to execute multimedia tasks and improve the execution efficiency of multimedia tasks.

[0071] Please see Figure 7 , Figure 7 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 70 stores program instructions 71 that can be executed by a processor. The program instructions 71 are used to implement the steps in any of the above embodiments of the task execution method based on multimedia data.

[0072] In the above scheme, the computer-readable storage medium 70 obtains fused features based on the encoding features of the target multimedia data and the embedding features of the prompt text. The prompt text describes the target task to be performed on the target multimedia data in natural language. The decoding result of the fused features after multiple rounds of autoregressive decoding is used as the execution result of performing the target task on the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially. In each round of autoregressive decoding: attention is calculated based on the output features of the previous decoding process to obtain the first hidden layer features for attention calculation. Based on the feature vector of the target word in the first hidden layer features, the target feedforward layer is selected from several candidate feedforward layers in the current decoding process, and the weight factor of the target feedforward layer is determined. Based on the weight factor of the target feedforward layer, the second hidden layer features after the first hidden layer features are weighted and processed by the target feedforward layer to obtain the third hidden layer features. Based on the first hidden layer features and the third hidden layer features, the output features of the current decoding process are obtained. The decoding result of this round of autoregressive decoding is predicted by the output features of the last decoding process in this round of autoregressive decoding. Therefore, based on the decoding results of each round of autoregressive decoding, the execution result is obtained. On the one hand, because the target feedforward layer is adaptively selected from several candidate feedforward layers for feature processing during the autoregressive decoding process, compared to feature processing through all candidate feedforward layers, it can control the required computational resources while expanding model parameters to meet the execution requirements of different multimedia tasks as much as possible. On the other hand, the selection of the target feedforward layer takes into account the feature information of both the target multimedia data and the prompt text. Thus, the target feedforward layer can be adaptively selected in different scenarios where the prompt text is the same but the target multimedia data is different, or in different scenarios where the target multimedia data is the same but the prompt text is different. Compared to selecting the target feedforward layer by only referring to the feature information of either the target multimedia data or the prompt text, it can further meet the execution requirements of different multimedia tasks. Furthermore, after selecting the target feedforward layer based on the feature vector of the target word and determining the weight factor of the target feedforward layer, the selected target feedforward layer and its determined weight factor are shared among all words in the first hidden layer features. Compared to selecting the target feedforward layer and determining its weight factor separately for each word in the first hidden layer features, it can significantly reduce the routing overhead in the inference process and help improve the execution efficiency of multimedia tasks. Therefore, while meeting the execution requirements of different multimedia tasks as much as possible, it can reduce the computing resources required to execute multimedia tasks and improve the execution efficiency of multimedia tasks.

[0073] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0074] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0075] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0076] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0077] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0078] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0079] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A task execution method based on multimedia data, characterized in that, include: Based on the encoding features of the target multimedia data and the embedding features of the prompt text, a fusion feature is obtained; wherein, the prompt text describes the target task to be performed on the target multimedia data in natural language, and the decoding result of the fusion feature after multiple rounds of autoregressive decoding is used as the execution result of performing the target task on the target multimedia data, and each round of autoregressive decoding includes multiple decoding processes executed sequentially; In each round of autoregressive decoding: attention is calculated based on the output features of the previous decoding process in the current decoding process to obtain the first hidden layer features of the attention calculation; based on the feature vector of the target word in the first hidden layer features, a target feedforward layer is selected from several candidate feedforward layers in the current decoding process and the weight factor of the target feedforward layer is determined; based on the weight factor of the target feedforward layer, the second hidden layer features after the first hidden layer features are processed by the target feedforward layer are weighted to obtain the third hidden layer features; and based on the first hidden layer features and the third hidden layer features, the output features of the current decoding process are obtained; wherein, the decoding result of the current round of autoregressive decoding is predicted by the output features of the last decoding process in the current round of autoregressive decoding, the feature vector of the target word contains the feature information of both the target multimedia data and the prompt text, the target word is the last word in the first hidden layer features, and when the current decoding process is the first decoding process in the current round of autoregressive decoding, the input features of the current decoding process are the fused features; The execution result is obtained based on the decoding results of the autoregressive decoding in each round.

2. The method according to claim 1, characterized in that, The step of selecting a target feedforward layer and determining the weight factor of the target feedforward layer from several candidate feedforward layers in the current decoding process based on the feature vector of the target word in the first hidden layer features includes: Based on the feature vector of the target word, a prediction is made to obtain the selection score of each of the several candidate feedforward layers; wherein, the selection score of the candidate feedforward layer represents the appropriateness of the candidate feedforward layer in processing the first hidden layer features; Based on the selection scores of the candidate feedforward layers, the target feedforward layer is selected from the candidate feedforward layers and the weight factor of the target feedforward layer is determined.

3. The method according to claim 2, characterized in that, The step of selecting the target feedforward layer and determining the weight factor of the target feedforward layer based on the selection scores of the candidate feedforward layers includes: The candidate feedforward layers are sorted according to their respective selection scores, and the candidate feedforward layer with the highest preset score is selected as the target feedforward layer. Based on the selection score of the target feedforward layer, the weight factor of the target feedforward layer is determined; wherein, the candidate feedforward layers that are not selected as the target feedforward layer are configured with a weight factor of 0.

4. The method according to claim 1, characterized in that, The weighting of the second hidden layer features after the first hidden layer features have been processed by the target feedforward layer, based on the weighting factors of the target feedforward layer, yields the third hidden layer features, including: Based on the weight factors of each of the target feedforward layers, the feature vector of the current word in the first hidden layer features after processing by the corresponding target feedforward layer is weighted and summed to obtain the weighted vector of the current word. The weighted vectors of each word in the first hidden layer feature are retrieved from contiguous memory and combined sequentially to obtain the third hidden layer feature.

5. The method according to claim 1, characterized in that, Before obtaining the output features of the current decoding process based on the first hidden layer features and the third hidden layer features, the method further includes: Based on the processing of the first hidden layer features by several shared feedforward layers in the current decoding process, the fourth hidden layer features are obtained after the first hidden layer features have been processed by each of the shared feedforward layers. The fifth hidden layer features are obtained by accumulating the fourth hidden layer features after processing each of the shared feedforward layers. The step of obtaining the output features of the current decoding process based on the first hidden layer features and the third hidden layer features includes: Based on the first hidden layer feature, the third hidden layer feature, and the fifth hidden layer feature, the output feature of the current decoding process is obtained by fusing them.

6. The method according to claim 1, characterized in that, After performing attention calculation on the output features of the previous decoding process based on the current decoding process to obtain the first hidden layer features of the attention calculation, and before selecting the target feedforward layer from several candidate feedforward layers in the current decoding process and determining the weight factor of the target feedforward layer based on the feature vector of the target word in the first hidden layer features, the method further includes: The current decoding process is detected to have either several candidate feedforward layers or only one feedforward layer. The step of selecting a target feedforward layer and determining the weight factor of the target feedforward layer from several candidate feedforward layers in the current decoding process based on the feature vector of the target word in the first hidden layer features includes: In response to the presence of the plurality of candidate feedforward layers in the current decoding process, the step of selecting a target feedforward layer and determining the weight factor of the target feedforward layer based on the feature vector of the target word in the first hidden layer features is executed.

7. The method according to claim 6, characterized in that, The method further includes: In response to the fact that there is only one feedforward layer in the current decoding process, the output features of the current decoding process are obtained by fusing the first hidden layer features and the sixth hidden layer features after the first hidden layer features are processed by the feedforward layer in the current decoding process.

8. The method according to claim 6, characterized in that, At least one of the plurality of decoding processes is selected at a target interval, wherein the decoding process is provided with the plurality of candidate feedforward layers.

9. The method according to any one of claims 1 to 8, characterized in that, The target multimedia data includes at least one of image data, audio data, video data, and animation data; And / or, the target task includes at least one of the following: image recognition task, text recognition task, audio recognition task, video analysis task, and screen expansion task; And / or, the execution result is obtained by a multimodal large model based on the target multimedia data and the prompt text. The multimodal large model includes an encoder, an embedding layer, and a decoder. The encoder is used to perform feature encoding on the target multimedia data, the embedding layer is used to perform text embedding on the prompt text, and the decoder is used to perform the autoregressive decoding.

10. A task execution device based on multimedia data, characterized in that, include: The fusion module is used to obtain fused features based on the encoding features of the target multimedia data and the embedding features of the prompt text; wherein, the prompt text describes the target task to be performed on the target multimedia data in natural language, and the decoding result of the fused features after multiple rounds of autoregressive decoding is used as the execution result of performing the target task on the target multimedia data, and each round of autoregressive decoding includes multiple decoding processes executed sequentially; The decoding module is used to: perform attention calculation based on the output features of the previous decoding process in each round of autoregressive decoding to obtain the first hidden layer features of the attention calculation; select a target feedforward layer from several candidate feedforward layers in the current decoding process based on the feature vector of the target word in the first hidden layer features and determine the weight factor of the target feedforward layer; weight the second hidden layer features after the first hidden layer features are processed by the target feedforward layer based on the weight factor of the target feedforward layer to obtain the third hidden layer features; and obtain the output features of the current decoding process based on the first hidden layer features and the third hidden layer features; wherein, the decoding result of the current round of autoregressive decoding is predicted by the output features of the last decoding process in the current round of autoregressive decoding, the feature vector of the target word contains the feature information of both the target multimedia data and the prompt text, the target word is the last word in the first hidden layer features, and when the current decoding process is the first decoding process in the current round of autoregressive decoding, the input features of the current decoding process are the fused features; The acquisition module is used to obtain the execution result based on the decoding results of the autoregressive decoding in each round.

11. An electronic device, characterized in that, It includes at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor is used to execute the program instructions to implement the task execution method based on multimedia data as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the task execution method based on multimedia data as described in any one of claims 1 to 9.

Images