Task execution method based on multimedia data, related device, equipment and medium

Abstract

The invention discloses a task execution method based on multimedia data, a related device, equipment and a medium, and the method comprises the steps: obtaining a fusion feature based on the coding feature of target multimedia data and the embedding feature of a prompt text; performing attention calculation based on the output feature of the previous decoding process to obtain a first hidden layer feature, selecting a target feed-forward layer from a plurality of candidate feed-forward layers in the current decoding process based on a feature vector of a target lexical element in the first hidden layer feature, and determining a weight factor of the target feed-forward layer, weighting a second hidden layer feature after the first hidden layer feature is processed on the basis of a weight factor of the target feed-forward layer to obtain a third hidden layer feature, and obtaining an output feature of the current decoding process on the basis of the first hidden layer feature and the third hidden layer feature; and obtaining an execution result based on a decoding result of each round of autoregressive decoding. According to the scheme, computing resources required for executing the multimedia task can be reduced, and the execution efficiency of the multimedia task is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a task execution method based on multimedia data and related devices, equipment and media. Background Art

[0002] With the rapid development of artificial intelligence technology, large multimodal models have gradually been widely used in many scenarios such as education and medical care.

[0003] Currently, to adapt large multimodal models to complex and diverse multimedia tasks, the number of model parameters in these models is exponentially increasing, leading to a corresponding surge in the demand for computing resources and, at the same time, impacting the efficiency of multimedia task execution. Therefore, how to reduce the computing resources required to execute multimedia tasks and improve their efficiency while maximally meeting the execution requirements of different multimedia tasks has become a pressing issue. Summary of the Invention

[0004] The main technical problem solved by this application is to provide a task execution method based on multimedia data and related devices, equipment and media, which can reduce the computing resources required to execute multimedia tasks and improve the execution efficiency of multimedia tasks while meeting the execution requirements of different multimedia tasks as much as possible.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a task execution method based on multimedia data, including: obtaining fusion features based on the encoding features of the target multimedia data and the embedded features of the prompt text; wherein the prompt text describes the target task to be performed on the target multimedia data in natural language, and the decoding results of the fusion features after multiple rounds of autoregressive decoding are used as the execution results of the target task on the target multimedia data, and each round of autoregressive decoding includes multiple decoding processes executed sequentially; in the process of each round of autoregressive decoding: attention calculation is performed based on the output features of the previous decoding process of the current decoding process to obtain attention A first hidden layer feature is calculated, and based on the feature vector of the target word in the first hidden layer feature, a target feed-forward layer is selected from several candidate feed-forward layers in the current decoding process and a weight factor of the target feed-forward layer is determined, and based on the weight factor of the target feed-forward layer, the second hidden layer feature after the first hidden layer feature is processed by the target feed-forward layer is weighted to obtain a third hidden layer feature, and based on the first hidden layer feature and the third hidden layer feature, an output feature of the current decoding process is obtained; wherein, the decoding result of this round of autoregressive decoding is predicted by the output feature of the last decoding process in this round of autoregressive decoding; and an execution result is obtained based on the decoding results of each round of autoregressive decoding.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a task execution device based on multimedia data, including: a fusion module, a decoding module, and an acquisition module. The fusion module is used to obtain fusion features based on the encoding features of the target multimedia data and the embedded features of the prompt text; wherein the prompt text describes the target task to be performed on the target multimedia data in natural language, and the decoding results of the fusion features after multiple rounds of autoregressive decoding are used as the execution results of the target task on the target multimedia data, and each round of autoregressive decoding includes multiple decoding processes executed in sequence; the decoding module is used to: perform a task based on the output features of the previous decoding process of the current decoding process in the process of each round of autoregressive decoding; Attention calculation, obtaining the first hidden layer feature of the attention calculation, and based on the feature vector of the target word in the first hidden layer feature, selecting the target feedforward layer from several candidate feedforward layers in the current decoding process and determining the weight factor of the target feedforward layer, based on the weight factor of the target feedforward layer, weighting the second hidden layer feature after the first hidden layer feature is processed by the target feedforward layer to obtain the third hidden layer feature, and based on the first hidden layer feature and the third hidden layer feature, obtaining the output feature of the current decoding process; wherein, the decoding result of this round of autoregressive decoding is predicted by the output feature of the last decoding process in this round of autoregressive decoding; an acquisition module is used to obtain the execution result based on the decoding results of each round of autoregressive decoding.

[0007] In order to solve the above technical problems, the third aspect of this application provides an electronic device, which at least includes a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor is used to execute the program instructions to implement the multimedia data-based task execution method in the above-mentioned first aspect.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the multimedia data-based task execution method of the above first aspect.

[0009] The above scheme obtains fusion features based on the encoding features of the target multimedia data and the embedded features of the prompt text. The prompt text describes the target task to be performed on the target multimedia data in natural language. The decoding results of the fusion features after multiple rounds of autoregressive decoding are used as the execution results of the target task on the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially. Therefore, in the process of each round of autoregressive decoding: attention calculation is performed based on the output features of the previous decoding process of the current decoding process to obtain the first hidden layer feature of the attention calculation, and based on the feature vector of the target word in the first hidden layer feature, the target feedforward layer is selected from several candidate feedforward layers in the current decoding process and the weight factor of the target feedforward layer is determined. The second hidden layer feature after the first hidden layer feature is processed by the target feedforward layer is weighted based on the weight factor of the target feedforward layer to obtain the third hidden layer feature, and the output feature of the current decoding process is obtained based on the first hidden layer feature and the third hidden layer feature. The decoding result of this round of autoregressive decoding is predicted by the output feature of the last decoding process in this round of autoregressive decoding, and then the execution result is obtained based on the decoding results of each round of autoregressive decoding. Therefore, on the one hand, due to the During the autoregressive decoding process, a target feed-forward layer is adaptively selected from several candidate feed-forward layers for feature processing. Compared to performing feature processing on all candidate feed-forward layers, this allows for controlling the required computing resources while expanding model parameters to best meet the execution requirements of different multimedia tasks. Furthermore, the target feed-forward layer is selected by simultaneously referencing the feature information of both the target multimedia data and the prompt text. This allows for adaptive selection of the target feed-forward layer in different scenarios with the same prompt text but different target multimedia data, or in different scenarios with the same target multimedia data but different prompt text. This further meets the execution requirements of different multimedia tasks compared to selecting the target feed-forward layer based solely on the feature information of either the target multimedia data or the prompt text. Furthermore, after selecting the target feed-forward layer based on the feature vector of the target word and determining its weight factor, the selected target feed-forward layer and its determined weight factor are shared across all word-units in the first hidden layer features. This significantly reduces routing overhead during inference, compared to selecting and determining the target feed-forward layer weight factor for each word-unit in the first hidden layer features, helping to improve the execution efficiency of multimedia tasks. Therefore, under the premise of satisfying the execution requirements of different multimedia tasks as much as possible, the computing resources required for executing multimedia tasks can be reduced and the execution efficiency of multimedia tasks can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 This is a flowchart of an embodiment of a method for executing a task based on multimedia data of the present application; Figure 2 This is a schematic diagram of the framework of an embodiment of the multimodal large model of the present application; Figure 3 It is a schematic diagram of an embodiment of the feature replacement operation of the present application; Figure 4 It is a schematic diagram of an embodiment of the feature rearrangement operation of the present application; Figure 5 This is a schematic diagram of a framework of an embodiment of a multimedia data-based task execution device of the present application; Figure 6 This is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The following describes the embodiments of the present application in detail with reference to the accompanying drawings.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] The terms "system" and "network" are often used interchangeably in this document. The term "and / or" is simply a description of an association between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the fragment " / " generally indicates that the related objects are in an "or" relationship. Furthermore, "multiple" in this document refers to two or more than two.

[0014] See also Figure 1 , Figure 1 This is a flowchart of an embodiment of the method for executing a task based on multimedia data of the present application. Specifically, the method may include the following steps: Step S11: obtaining fusion features based on the encoding features of the target multimedia data and the embedded features of the prompt text.

[0015] In the disclosed embodiment, the prompt text may describe the target task to be performed on the target multimedia data in natural language. For example, if the target task is relatively complex, the prompt text may describe the target task in a long sentence; or, if the target task is relatively simple, the prompt text may describe the target task in a short sentence.

[0016] In one implementation scenario, the target multimedia data may include, but is not limited to, image data, audio data, video data, animation data, and the like, and the specific types of the target multimedia data are not limited herein. It should be noted that although the target multimedia data is named "multimedia data", it does not limit the target multimedia data to contain two or more different types of data. The target multimedia data may also contain a single type of data. For example, the target multimedia data may contain only image data, or the target multimedia data may contain only audio data. Of course, the target multimedia data may also contain two or more different types of data. For example, the target multimedia data may be video data, in which case the target multimedia data contains audio data and image data, and may even contain text data (i.e., video subtitles). The above examples are merely a few possible examples of the target multimedia data, and the specific types of the target multimedia data are not limited herein, nor will they be given one by one.

[0017] In one implementation scenario, the target task may include, but is not limited to, image recognition tasks, text recognition tasks, audio recognition tasks, video analysis tasks, screen expansion tasks, and the like, and the specific type of the target task is not limited herein. For example, where the target multimedia data is image data and the target task is an image recognition task, the prompt text may include, but is not limited to, the following text content: "Please identify and output whether there is XX target in the image and, if there is XX target, the specific number of XX targets" (i.e., the image recognition task in this case is specifically to identify targets in the image), etc. Of course, the above examples are merely possible examples of the specific content of the prompt text when using image recognition tasks as an example in actual applications. Other possible scenarios are not limited herein, and no further examples are given.

[0018] In one implementation scenario, the encoded features of the target multimedia data can be obtained by encoding the features of the target multimedia data using an encoder. For example, the encoder may include, but is not limited to, a convolutional neural network, a variational autoencoder, a convolutional autoencoder, a visual Transformer encoder, or a cross-modal encoder such as CLIP (Contrastive Language-Image Pretraining). The network type of the encoder is not limited here, and examples are not given one by one.

[0019] In one implementation scenario, the embedded features of the prompt text can be obtained by embedding features in an embedding layer. For example, feature embedding can be implemented using methods including, but not limited to, statistical coding, word vector models, and pre-trained language models. The implementation methods of feature embedding are not limited here and will not be listed one by one.

[0020] In one implementation scenario, as a possible implementation example, after obtaining the coding features and the embedded features, they can be weighted based on the coding features and the embedded features to obtain fused features; or, as another possible implementation example, after obtaining the coding features and the embedded features, the coding features and the embedded features can also be processed based on the cross-attention mechanism to achieve feature interaction between the coding features and the embedded features, and then the output features after processing by the cross-attention mechanism can be used as fused features. It should be noted that the fused features can contain feature information of both the target multimedia data and the prompt text. In addition, the above examples are only a few possible examples of obtaining fused features of both the coding features and the embedded features, and other possible acquisition methods in actual applications are not limited here. For example, in actual applications, the fusion method of the coding features and the embedded features can be such that the fused features can contain feature information of both the target multimedia data and the prompt text.

[0021] It should be noted that in the disclosed embodiments, the decoding results of multiple rounds of autoregressive decoding, after fusion features, can serve as the execution results of the target task for the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially. The core concept of autoregressive decoding technology is to construct a complete sequence through a "word-by-word generation and cyclic dependency" approach. That is, the decoding result of each round of autoregressive decoding depends on the decoding results of previous rounds of autoregressive decoding until a termination condition is met (e.g., the decoding result of a certain round of autoregressive decoding reaches a terminator, the accumulated decoding results up to the current round of autoregressive decoding reach a maximum length, etc.). In the disclosed embodiments, to minimize the complexity of the solution description, the specific "word-by-word generation and cyclic dependency" approach in autoregressive decoding technology is described in detail. In practical applications, each round of autoregressive decoding can obtain the decoding results of previous rounds of autoregressive decoding and apply these decoding results to the current round of autoregressive decoding. For example, these decoding results can be used to extract feature vectors and fused with the encoding features of the target multimedia data and the embedded features of the prompt text to obtain fused features, which serve as the input features for the current round of autoregressive decoding. The specific process can be referred to the technical details of autoregressive decoding technology and will not be further described here. For easier understanding, please refer to Figure 2 , Figure 2 This is a schematic diagram of the framework of an embodiment of the multimodal large model of this application. Figure 2As shown, the multimodal large model may include an encoder, an embedding layer (not shown) and a decoder, wherein the encoder may be used to encode the target multimedia data, and the embedding layer may be used to embed the prompt text. It should be noted that the encoder and the embedding layer may refer to the aforementioned related descriptions. In addition, the decoder may be used to perform autoregressive decoding. Exemplarily, the decoder may include but is not limited to L-layer stacked Transformer modules, etc., and the network structure of the decoder is not limited here. Taking the decoder including L-layer stacked Transformer modules as an example, each round of autoregressive decoding needs to pass through L-layer stacked Transformer modules, and each decoding process in each round of autoregressive decoding corresponds to the processing process of different layers of Transformer modules (such as Figure 2 (As shown in the rectangular blocks filled with diagonal lines in the figure). For example, in the embodiments of the present disclosure, the Transformer module may include an attention layer and a feedforward layer. In practical applications, the Transformer module may also include other network layers, such as a normalization layer, which is not limited here.

[0022] Step S12: During each round of autoregressive decoding: perform attention calculation based on the output features of the previous decoding process of the current decoding process to obtain the first hidden layer feature of the attention calculation, and based on the feature vector of the target word in the first hidden layer feature, select the target feedforward layer from several candidate feedforward layers in the current decoding process and determine the weight factor of the target feedforward layer, weight the second hidden layer feature after the first hidden layer feature is processed by the target feedforward layer based on the weight factor of the target feedforward layer to obtain the third hidden layer feature, and obtain the output feature of the current decoding process based on the first hidden layer feature and the third hidden layer feature.

[0023] In the disclosed embodiment, the decoding result of this round of autoregressive decoding is predicted by the output features of the last decoding process in this round of autoregressive decoding. Exemplarily, the output features of the last decoding process in this round of autoregressive decoding can be predicted through network layers such as linear layers and softmax layers to obtain the decoding result of this round of autoregressive decoding. Still taking the decoder including L layers of stacked Transformer modules as an example, the last Transformer module in the decoder can be followed by a linear layer and a softmax layer, then the output features of the last Transformer module can be processed in sequence through the subsequent linear layer and softmax layer to predict the decoding result of this round of autoregressive decoding. For details, please refer to the technical details of the Transformer-based decoder, which will not be repeated here.

[0024] In one implementation scenario, for each decoding process in each round of autoregressive decoding, attention calculation can be first performed based on the output features of the previous decoding process of the current decoding process to obtain the first hidden layer features of the attention calculation. It should be noted that when the current decoding process is the first decoding process in this round of autoregressive decoding, since the current decoding process does not have a previous decoding process, the fused features can be directly used as the input features of the current decoding process, that is, attention calculation can be directly performed based on the fused features to obtain the first hidden layer features of the attention calculation. In addition, the attention calculation can be implemented by attention mechanisms including but not limited to self-attention mechanism, multi-head self-attention mechanism, etc. The technical details of the above-mentioned attention mechanisms are specifically referred to here. In the case that the current round of autoregressive decoding is not the first round, the feature vector can also be extracted based on the decoding results obtained in the previous rounds of autoregressive decoding, and fused together with the encoding features and embedded features to obtain the fused features as the input features of the current round of autoregressive decoding. For details, please refer to the above-mentioned related descriptions and will not be repeated here.

[0025] In one implementation scenario, after obtaining the first hidden layer features, a target feed-forward layer can be selected from several candidate feed-forward layers in the current decoding process based on the feature vector of the target word in the first hidden layer features, and a weight factor for the target feed-forward layer can be determined. It should be noted that different candidate feed-forward layers can be suitable for different tasks and data models. For example, at least one candidate feed-forward layer can be suitable for audio data recognition tasks, at least one candidate feed-forward layer can be suitable for image data recognition tasks, at least one candidate feed-forward layer can be suitable for audio data translation tasks, and at least one candidate feed-forward layer can be suitable for video data analysis tasks. Examples are not given here one by one. Furthermore, the network structure of a candidate feed-forward layer can be the same as (or different from) a standard feed-forward layer. The main difference between the candidate feed-forward layer and the standard feed-forward layer lies in the network parameters obtained by training with a specific training set, making it suitable for the application scenario represented by the specific training set. For example, a standard feed-forward layer can be trained based on sample image data for an image recognition task to obtain a candidate feed-forward layer, and the candidate feed-forward layer can then be suitable for image data recognition tasks. Candidate feed-forward layers suitable for other application scenarios can be trained in a similar manner and are not described here in detail.

[0026] In a specific implementation scenario, the target word can be the last word in the first hidden layer feature. For ease of understanding, taking the example of the first hidden layer containing s words and each word represented by a d-dimensional feature vector, the target word is the s-th word in the first hidden layer feature, and the feature vector of the target word is the d-dimensional feature vector of the s-th word in the first hidden layer feature. In the above method, by setting the target word as the last word in the first hidden layer feature, historical information can be reviewed to assist autoregressive decoding.

[0027] In a specific implementation scenario, after obtaining the feature vector of the target word in the first hidden layer feature, prediction can be made based on the feature vector of the target word to obtain the selection scores of several candidate feed-forward layers. For example, the feature vector of the target word can be predicted by the fully connected layer to obtain the selection scores of several candidate feed-forward layers. It should be noted that the selection score of the candidate feed-forward layer can be characterized as the suitability of the candidate feed-forward layer for processing the first hidden layer feature. For example, the higher the selection score of the candidate feed-forward layer, the higher the suitability of the candidate feed-forward layer for processing the first hidden layer feature. Conversely, the lower the selection score of the candidate feed-forward layer, the lower the suitability of the candidate feed-forward layer for processing the first hidden layer feature. On this basis, based on the selection scores of several candidate feed-forward layers, a target feed-forward layer can be selected from several candidate feed-forward layers and the weight factor of the target feed-forward layer can be determined. Specifically, several candidate feedforward layers can be sorted based on their respective selection scores, and the candidate feedforward layer with a preset high position (such as top2, top3, etc.) can be selected as the target feedforward layer. Then, based on the selection score of the target feedforward layer, the weight factor of the target feedforward layer can be determined, and the candidate feedforward layer that is not selected as the target feedforward layer is configured with a weight factor of 0. For example, the candidate feedforward layers can be sorted in descending order according to the selection score, and the candidate feedforward layer before the preset position (such as the top 2, top 3, etc.) can be selected as the target feedforward layer. Based on this, the selection score of the target feedforward layer can be used as the weight factor of the target feedforward layer, or it can be normalized based on the selection scores of each target feedforward layer to obtain the weight factor of the target feedforward layer. Of course, the above examples are only a few possible examples of obtaining weight factors. The measurement method of the weight factor is not limited here, and no examples are given one by one. For ease of understanding, taking the lth decoding process as an example, the feature vector of the target word (i.e., the sth word) in the first hidden layer feature is recorded as For example, the selection score s of the i-th candidate feedforward layer is i It can be expressed as:

[0028] In the above formula, f c represents the fully connected layer, and softmax represents the softmax layer. On this basis, the weight factors of each candidate feedforward layer can be obtained, where the weight factor g of the i-th candidate feedforward layer is i It can be expressed as:

[0029] In the above formula, Topk represents the selection score s1 of the first candidate feedforward layer to the selection score s of the Nth candidate feedforward layer. NThe K candidate feedforward layers with the highest scores are selected as the target feedforward layers, and their weight factors are configured as their selection scores. That is, in this case, the weight factor g of the i-th target feedforward layer is i That is, select the score s for it i itself. In addition, except for the above-mentioned K candidate feed-forward layers, the weight factors of other candidate feed-forward layers are configured to be 0. The above method performs prediction based on the feature vector of the target word to obtain the selection scores of several candidate feed-forward layers, and the selection scores of the candidate feed-forward layers are characterized as the suitability of the candidate feed-forward layer for processing the first hidden layer features. Based on the selection scores of the several candidate feed-forward layers, the target feed-forward layer is selected from the several candidate feed-forward layers and the weight factor of the target feed-forward layer is determined, which can maximize the adaptability of the target feed-forward layer to the target multimedia data and its target task.

[0030] In one implementation scenario, after determining the target feedforward layer and its weight factor, the feature vector of the current word in the first hidden layer feature after being processed by the corresponding target feedforward layer can be weighted and summed based on the weight factors of each target feedforward layer to obtain the weighted vector of the current word. On this basis, the weighted vectors of each word in the first hidden layer feature can be taken out from the continuous memory and combined in sequence to obtain the third hidden layer feature. In the above method, each word in the first hidden layer feature shares the target feedforward layer and its weight factor, which can save routing overhead as much as possible compared to selecting the target feedforward layer and determining the weight factor for each word separately, which helps to improve the reasoning efficiency of task execution.

[0031] In a specific implementation scenario, each word in the first hidden layer feature can be used as the current word, and then the feature vector of the current word can be processed by each target feed-forward layer. After the processing is completed, the feature vectors after the corresponding target feed-forward layer processing can be weighted and summed using the weight factors of each target feed-forward layer to obtain the weighted vector of the current word. Finally, the weighted vectors of each word in the first hidden layer feature can be retrieved from the continuous memory and combined in sequence to obtain the third hidden layer feature. For example, the weighted vector of the first word, the weighted vector of the second word, ..., the weighted vector of the sth word are combined in sequence to obtain the third hidden layer feature.

[0032] In a specific implementation scenario, for the convenience of description, please continue to refer to Figure 2 , still taking the lth decoding process as an example, for the tth word in the first hidden layer feature, its feature vector can be recorded as , then its weight vector can be expressed as: Among them, g i Represents each candidate feed-forward layer (such as Figure 2As mentioned above, the weight factor of the candidate feedforward layer that is not selected as the target feedforward layer is configured as 0. For this part of the candidate feedforward layer, it is equivalent to not using it to process the word element vector. i represents the i-th candidate feed-forward layer, and N represents the total number of candidate feed-forward layers.

[0033] In one implementation scenario, as a possible implementation method, after obtaining the first hidden layer feature and the third hidden layer feature, the first hidden layer feature and the third hidden layer feature can be fused to obtain the output feature of the current decoding process. For example, the first hidden layer feature and the third hidden layer feature can be summed to obtain the output feature of the current decoding process. Alternatively, as another possible implementation method, please continue to refer to Figure 2 , several shared feedforward layers can also be configured in the current decoding process (such as Figure 2 ), the first hidden layer features can be processed separately based on several shared feedforward layers in the current decoding process to obtain the fourth hidden layer features after the first hidden layer features are processed by each shared feedforward layer, and then the fourth hidden layer features after each shared feedforward layer are accumulated to obtain the fifth hidden layer features. On this basis, the output features of the current decoding process can be obtained by fusing the first hidden layer features, the third hidden layer features and the fifth hidden layer features. Exemplarily, the first hidden layer features, the third hidden layer features and the fifth hidden layer features can be summed up as the output features of the current decoding process. For ease of description, please continue to refer to Figure 2 , still taking the lth decoding process as an example, for the tth word in the first hidden layer feature, its feature vector can be recorded as , then the feature vector after being processed by each shared feedforward layer and then accumulated can be expressed as: Among them, FNN i represents the i-th shared feedforward layer, K s Represents the total number of shared feedforward layers. Furthermore, for the t-th word in the first hidden layer feature, its output vector after passing through the candidate feedforward layer and the shared feedforward layer is It can be expressed as:

[0034] From the above formula, we can see that the feature vector of the t-th word in the output feature of the current decoding process is the above output feature . In other words, the output features of each word in the first hidden layer feature after the above processing can be combined to obtain the output features of the current decoding process. Similar to the candidate feedforward layer, the network structure of the shared feedforward layer can also refer to the standard feedforward layer, which will not be repeated here. In addition, unlike the candidate feedforward layer, all inputs are "globally shared" in the shared feedforward layer without routing any part of it, that is, the shared feedforward layer does not rely on the routing mechanism. It is mainly responsible for capturing the general and common feature information in the input features (such as basic grammatical rules, low-level visual features, etc.). For details, please refer to the technical details of the shared expert, which will not be repeated here.

[0035] It should be noted that the above implementation process is the specific process of outputting features of the current decoding process. In practical applications, each decoding process in each round of autoregressive decoding can be equipped with a candidate feedforward layer, and even a shared feedforward layer can be further equipped. Alternatively, at least one decoding process in each round of autoregressive decoding can be selected at a target interval to be equipped with a candidate feedforward layer. For example, a candidate feedforward layer can be equipped in every other decoding process, and a shared feedforward layer can also be equipped at the same time as the candidate feedforward layer. Of course, only a candidate feedforward layer can also be equipped, and this is not limited here. In order to deal with the above situation, after obtaining the first hidden layer feature and before selecting the target feedforward layer and determining its weight factor, it is also possible to first detect whether there are several candidate feedforward layers or only one feedforward layer in the current decoding process. In response to the fact that there are several candidate feedforward layers in the current decoding process, the steps of selecting the target feedforward layer from several candidate feedforward layers in the current decoding process and determining the weight factor of the target feedforward layer can be executed based on the feature vector of the target word in the first hidden layer feature. For details, please refer to the above-mentioned relevant description and will not be repeated here. On the contrary, in response to the fact that there is only one feedforward layer in the current decoding process, the first hidden layer feature and the sixth hidden layer feature after the first hidden layer feature is processed by the feedforward layer in the current decoding process can be fused (such as, added, etc.) to obtain the output feature of the current decoding process.

[0036] In order to facilitate understanding of the effect of saving routing overhead by the above decoding process in the embodiment of the present disclosure, another related implementation method is introduced below as a comparative example of the above decoding process in the embodiment of the present disclosure. Different from the decoding process in the embodiment of the present disclosure, in the current decoding process of the comparative example, for the feature vector of each word in the first hidden layer feature, the following process steps can be repeatedly performed: based on the feature vector of the current word, prediction is performed to obtain the selection scores of several candidate feedforward layers, and based on the selection scores of several candidate feedforward layers, a target feedforward layer is selected from several candidate feedforward layers and the weight factor of the target feedforward layer is determined, based on the weight factor of the target feedforward layer, the feature vector after the feature vector is processed by the target feedforward layer is weighted and summed to obtain the weighted vector of the current word, and based on the weighted vector of the current word and the feature vector of the current word, fusion (such as addition, etc.) is performed to obtain the output vector of the current word in the current decoding process, and the output feature of the current decoding process is obtained by combining the output vectors of each word. In order to facilitate the understanding of the comparative example and to facilitate the comparison with the above decoding process of the embodiment of the present disclosure, the feature vector of the t-th word in the first hidden layer feature in the l-th decoding process is still used. For example, the selection score s of the i-th candidate feedforward layer i,t It can be expressed as:

[0037] On this basis, the weight factors of each candidate feedforward layer can be determined, where the weight factor g of the i-th candidate feedforward layer when the t-th word is processed in the decoding process is i,t It can be expressed as:

[0038] On this basis, the output vector of the t-th word in the first hidden layer feature is It can be expressed as:

[0039] Of course, when there is a shared feedforward layer in the lth decoding process, the output vector of the tth word in the first hidden layer feature is It can be expressed as:

[0040] It can be seen that the main difference between the decoding process in the comparative example and the embodiment of the present disclosure is that the target feedforward layer and its weight factor in the comparative example are not shared by all the word units of the first hidden layer feature, but each word unit determines the target feedforward layer and its weight factor separately. Figure 3 and Figure 4 , Figure 3This is a schematic diagram of an embodiment of the feature replacement operation of the present application. Figure 4 This is a schematic diagram of an embodiment of the feature rearrangement operation of this application. Figure 3 As shown, taking the example of the first hidden layer feature containing 5 words (respectively recorded as word 0, word 1, word 2, word 3, and word 4) and a total of 4 candidate feed-forward layers (respectively recorded as candidate feed-forward layer 0, candidate feed-forward layer 1, candidate feed-forward layer 2, and candidate feed-forward layer 3), their respective feature vectors are indicated by rectangular boxes filled with different shades. If the configuration selects the top two digits of the score as the target feed-forward layer, then word 0 is predicted to determine candidate feed-forward layer 1 and candidate feed-forward layer 3 as the target feed-forward layer of word 0, word 1 is predicted to determine candidate feed-forward layer 0 and candidate feed-forward layer 3 as the target feed-forward layer of word 1, word 2 is predicted to determine candidate feed-forward layer 0 and candidate feed-forward layer 1 as the target feed-forward layer of word 2, word 3 is predicted to determine candidate feed-forward layer 1 and candidate feed-forward layer 3 as the target feed-forward layer of word 3, and word 4 is predicted to determine candidate feed-forward layer 1 and candidate feed-forward layer 1 as the target feed-forward layer of word 4. On this basis, the feature vectors that need to be processed by the same feedforward layer are aggregated, that is, the candidate feedforward layer 0 aggregates the feature vectors of word 1, word 2 and word 4 for processing, the candidate feedforward layer 1 aggregates the feature vectors of word 0, word 2, word 3 and word 4 for processing, the candidate feedforward layer 3 aggregates the feature vectors of word 0, word 1 and word 3 for processing, and the candidate feedforward layer 2 has no feature vectors that need to be processed. After the calculation is completed, it is necessary to execute the following Figure 4 The rearrangement operation shown in , passes the calculation results of each candidate feed-forward layer back to the original sample, and reorganizes them according to the original order of each word. Figure 4 In the candidate feedforward layer 0, after processing the feature vectors of word 1, word 2 and word 4 respectively, the three weight factors (such as Figure 4 The candidate feed-forward layer 0 needs to process the feature vectors of word 1, word 2 and word 4, as shown by the numbers 0.2, 0.9 and 0.7 after each processing. Similarly, after the candidate feed-forward layer 1 processes the feature vectors of word 0, word 2, word 3 and word 4 respectively, it can continue to use their respective weight factors (such as Figure 4 The candidate feed-forward layer 1 needs to process the feature vectors of word 0, word 2, word 3 and word 4, as shown by the numbers 0.3, 0.1, 0.4 and 0.3 after the feature vectors are multiplied by the processed feature vectors; similarly, after the candidate feed-forward layer 3 processes the feature vectors of word 0, word 1 and word 3 respectively, it can continue to use their respective weight factors (such as Figure 4The feature vectors for the candidate feedforward layer 3 (denoted by the numbers 0.7, 0.8, and 0.6 after the feature vectors of tokens 0, 1, and 3) are multiplied by the processed feature vectors. Based on this, the feature vectors, weighted by the weighting factors, are reorganized into the original order of tokens 0, 1, 2, and 3. For example, for word 0, its feature vectors after being weighted by candidate feedforward layer 1 and candidate feedforward layer 3 can be combined to reorganize and obtain the weighted vector of word 0; similarly, for word 1, its feature vectors after being weighted by candidate feedforward layer 0 and candidate feedforward layer 3 can be combined to reorganize and obtain the weighted vector of word 1; similarly, for word 2, its feature vectors after being weighted by candidate feedforward layer 1 and candidate feedforward layer 3 can be combined to reorganize and obtain the weighted vector of word 2; similarly, for word 3, its feature vectors after being weighted by candidate feedforward layer 0 and candidate feedforward layer 1 can be combined to reorganize and obtain the weighted vector of word 3. Combine Figure 3 and Figure 4 As shown, in the feature permutation operation of the comparative example, it is necessary to group the word elements according to the candidate feedforward layer and read data non-continuously from the continuous memory (e.g., word elements 1, 2, 4 -> candidate feedforward layer 0). In the rearrangement operation of the comparative example, the calculation results of the candidate feedforward layer need to be reorganized in the original order. At this time, random writing to the memory is required, and the random access bandwidth of the GPU video memory is only 1 / 10 to 1 / 5 of the continuous access. Due to this hardware limitation, compared with the decoding process in the embodiment of the present disclosure, the use of the comparative example will increase the delay and fail to fully utilize the network bandwidth, thereby resulting in low execution efficiency of multimedia tasks. In other words, the decoding process in the embodiment of the present disclosure shares the target feedforward layer and its weight factor between the various word elements of the first hidden layer feature, and the overall overhead ratio of a single route can be reduced as the degree of autoregressive decoding increases. In addition, since the decoding process in the embodiment of the present disclosure does not disrupt the word element order, it is only necessary to copy the entire block of continuous memory during the feature permutation operation, and the memory will not be randomly written during the reorganization operation, so the memory can be accessed continuously. In particular, in the embodiment of the present disclosure, when the number of batches is 1, the feature replacement operation and the reorganization operation can be omitted in the decoding process.

[0041] Step S13: Obtaining an execution result based on the decoding results of each round of autoregressive decoding.

[0042] Specifically, at the end of autoregressive decoding, the decoding results of each round of autoregressive decoding can be sequentially combined to obtain an execution result. For example, when the target task is a text recognition task for image data, each round of autoregressive decoding can obtain a decoded character as the decoding result of that round of autoregressive decoding, and the decoded characters of each round of autoregressive decoding can then be sequentially combined to obtain the execution result of the text recognition task (i.e., the text content in the image data); or, when the target task is an audio recognition task for audio data, each round of autoregressive decoding can obtain a decoded character as the decoding result of that round of autoregressive decoding, and the decoded characters of each round of autoregressive decoding can then be sequentially combined to obtain the execution result of the audio recognition task (i.e., the recognized text of the audio data). It should be noted that the above examples are merely a few possible examples of execution results when the target multimedia data is of different data types and the target tasks are different tasks. Other possible scenarios are not limited here and will not be given one by one.

[0043] The above scheme obtains fusion features based on the encoding features of the target multimedia data and the embedded features of the prompt text. The prompt text describes the target task to be performed on the target multimedia data in natural language. The decoding results of the fusion features after multiple rounds of autoregressive decoding are used as the execution results of the target task on the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially. Therefore, in the process of each round of autoregressive decoding: attention calculation is performed based on the output features of the previous decoding process of the current decoding process to obtain the first hidden layer feature of the attention calculation, and based on the feature vector of the target word in the first hidden layer feature, the target feedforward layer is selected from several candidate feedforward layers in the current decoding process and the weight factor of the target feedforward layer is determined. The second hidden layer feature after the first hidden layer feature is processed by the target feedforward layer is weighted based on the weight factor of the target feedforward layer to obtain the third hidden layer feature, and the output feature of the current decoding process is obtained based on the first hidden layer feature and the third hidden layer feature. The decoding result of this round of autoregressive decoding is predicted by the output feature of the last decoding process in this round of autoregressive decoding, and then the execution result is obtained based on the decoding results of each round of autoregressive decoding. Therefore, on the one hand, due to the During the autoregressive decoding process, a target feed-forward layer is adaptively selected from several candidate feed-forward layers for feature processing. Compared to performing feature processing on all candidate feed-forward layers, this allows for controlling the required computing resources while expanding model parameters to best meet the execution requirements of different multimedia tasks. Furthermore, the target feed-forward layer is selected by simultaneously referencing the feature information of both the target multimedia data and the prompt text. This allows for adaptive selection of the target feed-forward layer in different scenarios with the same prompt text but different target multimedia data, or in different scenarios with the same target multimedia data but different prompt text. This further meets the execution requirements of different multimedia tasks compared to selecting the target feed-forward layer based solely on the feature information of either the target multimedia data or the prompt text. Furthermore, after selecting the target feed-forward layer based on the feature vector of the target word and determining its weight factor, the selected target feed-forward layer and its determined weight factor are shared across all word-units in the first hidden layer features. This significantly reduces routing overhead during inference, compared to selecting and determining the target feed-forward layer weight factor for each word-unit in the first hidden layer features, helping to improve the execution efficiency of multimedia tasks. Therefore, under the premise of satisfying the execution requirements of different multimedia tasks as much as possible, the computing resources required for executing multimedia tasks can be reduced and the execution efficiency of multimedia tasks can be improved.

[0044] See also Figure 5 , Figure 5This is a schematic diagram of the framework of an embodiment of a task execution device based on multimedia data of the present application. The task execution device based on multimedia data 50 includes: a fusion module 51, a decoding module 52 and an acquisition module 53. The fusion module 51 is used to obtain fusion features based on the encoding features of the target multimedia data and the embedded features of the prompt text; wherein the prompt text describes the target task to be performed on the target multimedia data in natural language, and the decoding results of the fusion features after multiple rounds of autoregressive decoding are used as the execution results of the target task on the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially; the decoding module 52 is used in the process of each round of autoregressive decoding: based on the output features of the previous decoding process of the current decoding process, to obtain To the first hidden layer feature of the attention calculation, and based on the feature vector of the target word in the first hidden layer feature, select the target feed-forward layer from several candidate feed-forward layers in the current decoding process and determine the weight factor of the target feed-forward layer, based on the weight factor of the target feed-forward layer, weight the second hidden layer feature after the first hidden layer feature is processed by the target feed-forward layer to obtain the third hidden layer feature, and based on the first hidden layer feature and the third hidden layer feature, obtain the output feature of the current decoding process; wherein, the decoding result of this round of autoregressive decoding is predicted by the output feature of the last decoding process in this round of autoregressive decoding; an acquisition module 53 is used to obtain an execution result based on the decoding results of each round of autoregressive decoding.

[0045] In the above-mentioned scheme, the multimedia data-based task execution device 50 obtains a fusion feature based on the encoding feature of the target multimedia data and the embedded feature of the prompt text. The prompt text describes the target task to be performed on the target multimedia data in natural language. The decoding result of the fusion feature after multiple rounds of autoregressive decoding is used as the execution result of performing the target task on the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially, so that in the process of each round of autoregressive decoding: attention calculation is performed based on the output feature of the previous decoding process of the current decoding process to obtain the first hidden layer feature of the attention calculation, and based on the feature vector of the target word in the first hidden layer feature, the target feedforward layer is selected from several candidate feedforward layers in the current decoding process and the weight factor of the target feedforward layer is determined, the second hidden layer feature after the first hidden layer feature is processed by the target feedforward layer is weighted based on the weight factor of the target feedforward layer to obtain the third hidden layer feature, and the output feature of the current decoding process is obtained based on the first hidden layer feature and the third hidden layer feature, and the decoding result of this round of autoregressive decoding is predicted by the output feature of the last decoding process in this round of autoregressive decoding, and then the execution result is obtained based on the decoding results of each round of autoregressive decoding. Therefore, on the one hand, since the target feed-forward layer is adaptively selected from several candidate feed-forward layers for feature processing during the autoregressive decoding process, compared with performing feature processing through all candidate feed-forward layers, the required computing resources can be controlled when expanding model parameters to meet the execution requirements of different multimedia tasks as much as possible. On the other hand, the feature information of both the target multimedia data and the prompt text is simultaneously referenced when selecting the target feed-forward layer. In this way, the target feed-forward layer can be adaptively selected in different scenarios with the same prompt text but different target multimedia data, or in different scenarios with the same target multimedia data but different prompt text. Compared with selecting the target feed-forward layer by only referring to the feature information of either the target multimedia data or the prompt text, the execution requirements of different multimedia tasks can be further met. On the other hand, after selecting the target feed-forward layer based on the feature vector of the target word and determining the weight factor of the target feed-forward layer, the selected target feed-forward layer and the determined weight factor are shared among all word-grams in the first hidden layer feature. Compared with selecting the target feed-forward layer and determining its weight factor for all word-grams in the first hidden layer feature, the routing overhead in the inference process can be significantly reduced, which helps to improve the execution efficiency of multimedia tasks. Therefore, under the premise of satisfying the execution requirements of different multimedia tasks as much as possible, the computing resources required for executing multimedia tasks can be reduced and the execution efficiency of multimedia tasks can be improved.

[0046] In some disclosed embodiments, the decoding module 52 includes a score prediction submodule for making predictions based on the feature vector of the target word to obtain selection scores for each of the candidate feed-forward layers; wherein the selection scores of the candidate feed-forward layers represent the suitability of the candidate feed-forward layer for processing the first hidden layer features; the decoding module 52 includes a selection determination submodule for selecting a target feed-forward layer from the candidate feed-forward layers and determining a weight factor of the target feed-forward layer based on the selection scores of the candidate feed-forward layers.

[0047] In some disclosed embodiments, the selection determination submodule includes a sorting selection unit for sorting several candidate feedforward layers based on their respective selection scores, and selecting a preset high-ranking candidate feedforward layer as the target feedforward layer; the selection determination submodule includes a weight determination unit for determining a weight factor of the target feedforward layer based on the selection score of the target feedforward layer; wherein, the candidate feedforward layer that is not selected as the target feedforward layer is configured with a weight factor of 0.

[0048] In some disclosed embodiments, the decoding module 52 includes a weighted summation submodule for performing weighted summation on the feature vectors of the current word unit in the first hidden layer features after being processed by the corresponding target feedforward layers based on the weight factors of each target feedforward layer to obtain the weighted vector of the current word unit; the decoding module 52 includes a memory reading submodule for taking out the weighted vectors of each word unit in the first hidden layer features from the continuous memory and combining them in sequence to obtain the third hidden layer features.

[0049] In some disclosed embodiments, the decoding module 52 includes a shared feedforward submodule for processing the first hidden layer features based on several shared feedforward layers in the current decoding process, respectively, to obtain the fourth hidden layer features after the first hidden layer features are processed by each shared feedforward layer; the decoding module 52 includes a feature accumulation submodule for accumulating the fourth hidden layer features after processing by each shared feedforward layer, to obtain the fifth hidden layer features; the decoding module 52 includes a hidden layer fusion submodule for fusing the first hidden layer features, the third hidden layer features and the fifth hidden layer features to obtain the output features of the current decoding process.

[0050] In some disclosed embodiments, the decoding module 52 includes a feedforward detection submodule for detecting whether there are several candidate feedforward layers or only one feedforward layer in the current decoding process; the decoding module 52 includes a first response submodule for responding to the fact that there are several candidate feedforward layers in the current decoding process, executing the steps of selecting a target feedforward layer from several candidate feedforward layers in the current decoding process based on the feature vector of the target word in the first hidden layer feature and determining the weight factor of the target feedforward layer.

[0051] In some disclosed embodiments, the decoding module 52 includes a second response submodule for obtaining the output features of the current decoding process based on the first hidden layer features and the sixth hidden layer features after the first hidden layer features are processed by the feedforward layer in the current decoding process in response to the fact that there is only one feedforward layer in the current decoding process.

[0052] In some disclosed embodiments, at least one decoding process is selected from a plurality of decoding processes at a target interval and is provided with a plurality of candidate feed-forward layers.

[0053] In some disclosed embodiments, the target multimedia data includes at least one of image data, audio data, video data, and animation data; and / or the target task includes at least one of an image recognition task, a text recognition task, an audio recognition task, a video analysis task, and a picture expansion task; and / or the target word is the last word in the first hidden layer feature; and / or the execution result is obtained by a multimodal large model based on the target multimedia data and the prompt text, and the multimodal large model includes an encoder, an embedding layer, and a decoder, the encoder is used to perform feature encoding on the target multimedia data, the embedding layer is used to perform text embedding on the prompt text, and the decoder is used to perform autoregressive decoding.

[0054] See also Figure 6 , Figure 6 : This is a schematic diagram of the framework of an embodiment of an electronic device of the present application. The electronic device 60 includes at least a memory 61 and a processor 62 coupled to each other. The memory 61 stores at least program instructions, and the processor 62 is used to execute the program instructions to implement the steps in any of the above-mentioned embodiments of the task execution method based on multimedia data. For details, please refer to the aforementioned disclosed embodiments and will not be repeated here. The electronic device 60 may include but is not limited to mobile phones, tablet computers, learning machines, translation machines, smart large screens, servers and other devices. The specific type of the electronic device 60 is not limited here.

[0055] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps of any of the above-mentioned embodiments of the multimedia data-based task execution method. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The general-purpose processor can be a microprocessor or any conventional processor. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0056] In the above scheme, the electronic device 60 obtains a fusion feature based on the encoding feature of the target multimedia data and the embedded feature of the prompt text. The prompt text describes the target task to be performed on the target multimedia data in natural language. The decoding result of the fusion feature after multiple rounds of autoregressive decoding is used as the execution result of performing the target task on the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially. Therefore, in the process of each round of autoregressive decoding: attention calculation is performed based on the output feature of the previous decoding process of the current decoding process to obtain the first hidden layer feature of the attention calculation, and based on the feature vector of the target word in the first hidden layer feature, the target feedforward layer is selected from several candidate feedforward layers in the current decoding process and the weight factor of the target feedforward layer is determined. The second hidden layer feature after the first hidden layer feature is processed by the target feedforward layer is weighted based on the weight factor of the target feedforward layer to obtain the third hidden layer feature, and the output feature of the current decoding process is obtained based on the first hidden layer feature and the third hidden layer feature. The decoding result of this round of autoregressive decoding is predicted by the output feature of the last decoding process in this round of autoregressive decoding, and then the execution result is obtained based on the decoding results of each round of autoregressive decoding. Therefore, on the one hand, Because a target feed-forward layer is adaptively selected from several candidate feed-forward layers for feature processing during the autoregressive decoding process, compared to performing feature processing through all candidate feed-forward layers, the required computing resources can be controlled while expanding model parameters to best meet the execution requirements of different multimedia tasks. Furthermore, by simultaneously referencing the feature information of both the target multimedia data and the prompt text when selecting the target feed-forward layer, the target feed-forward layer can be adaptively selected in different scenarios with the same prompt text but different target multimedia data, or in different scenarios with the same target multimedia data but different prompt text. This can further meet the execution requirements of different multimedia tasks compared to selecting the target feed-forward layer based on the feature vector of the target word and determining its weight factor. Furthermore, after selecting the target feed-forward layer based on the feature vector of the target word and determining its weight factor, the selected target feed-forward layer and its determined weight factor are shared across all word-units in the first hidden layer features. This significantly reduces routing overhead during inference, compared to selecting the target feed-forward layer and determining its weight factor for each word-unit in the first hidden layer features, helping to improve the execution efficiency of multimedia tasks. Therefore, under the premise of satisfying the execution requirements of different multimedia tasks as much as possible, the computing resources required for executing multimedia tasks can be reduced and the execution efficiency of multimedia tasks can be improved.

[0057] See also Figure 7 , Figure 7 The computer-readable storage medium 70 stores program instructions 71 that can be executed by a processor, and the program instructions 71 are used to implement the steps of any of the above-mentioned multimedia data-based task execution method embodiments.

[0058] In the above scheme, the computer-readable storage medium 70 obtains a fusion feature based on the encoding feature of the target multimedia data and the embedded feature of the prompt text. The prompt text describes the target task to be performed on the target multimedia data in natural language. The decoding result of the fusion feature after multiple rounds of autoregressive decoding is used as the execution result of performing the target task on the target multimedia data. Each round of autoregressive decoding includes multiple decoding processes executed sequentially. Therefore, in the process of each round of autoregressive decoding: attention calculation is performed based on the output feature of the previous decoding process of the current decoding process to obtain the first hidden layer feature of the attention calculation, and based on the feature vector of the target word in the first hidden layer feature, a target feedforward layer is selected from several candidate feedforward layers in the current decoding process and a weight factor of the target feedforward layer is determined. Based on the weight factor of the target feedforward layer, the second hidden layer feature after the first hidden layer feature is processed by the target feedforward layer is weighted to obtain a third hidden layer feature, and based on the first hidden layer feature and the third hidden layer feature, the output feature of the current decoding process is obtained, and the decoding result of this round of autoregressive decoding is predicted by the output feature of the last decoding process in this round of autoregressive decoding, and then the execution result is obtained based on the decoding results of each round of autoregressive decoding. Therefore, On the one hand, because a target feed-forward layer is adaptively selected from several candidate feed-forward layers for feature processing during the autoregressive decoding process, compared to performing feature processing through all candidate feed-forward layers, the required computing resources can be controlled while expanding model parameters to meet the execution requirements of different multimedia tasks as much as possible. On the other hand, the feature information of both the target multimedia data and the prompt text is simultaneously referenced when selecting the target feed-forward layer. In this way, the target feed-forward layer can be adaptively selected in different scenarios with the same prompt text but different target multimedia data, or in different scenarios with the same target multimedia data but different prompt text. Compared to selecting the target feed-forward layer based on only the feature information of either the target multimedia data or the prompt text, this can further meet the execution requirements of different multimedia tasks. On the other hand, after selecting the target feed-forward layer based on the feature vector of the target word and determining the weight factor of the target feed-forward layer, the selected target feed-forward layer and its determined weight factor are shared among all word-grams in the first hidden layer features. Compared with selecting the target feed-forward layer and determining its weight factor for each word-gram in the first hidden layer features, this can significantly reduce the routing overhead during inference, helping to improve the execution efficiency of multimedia tasks. Therefore, under the premise of satisfying the execution requirements of different multimedia tasks as much as possible, the computing resources required for executing multimedia tasks can be reduced and the execution efficiency of multimedia tasks can be improved.

[0059] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.

[0060] The above description of the various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced with each other and will not be repeated herein for the sake of brevity.

[0061] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0062] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0063] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0064] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the various implementation methods of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0065] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

Claims

1. A task execution method based on multimedia data, characterized in that: include: A fused feature is obtained based on the encoding features of the target multimedia data and the embedded features of the prompt text; wherein the prompt text describes a target task to be performed on the target multimedia data in natural language, and the decoding results of the fused feature after multiple rounds of autoregressive decoding are used as the execution results of the target task on the target multimedia data, wherein each round of the autoregressive decoding includes multiple decoding processes executed sequentially; During each round of the autoregressive decoding process: performing an attention calculation based on the output features of the previous decoding process of the current decoding process to obtain a first hidden layer feature of the attention calculation; and based on the feature vector of the target word in the first hidden layer feature, selecting a target feedforward layer from a number of candidate feedforward layers in the current decoding process and determining a weight factor of the target feedforward layer; weighting a second hidden layer feature after the first hidden layer feature is processed by the target feedforward layer based on the weight factor of the target feedforward layer to obtain a third hidden layer feature; and obtaining an output feature of the current decoding process based on the first hidden layer feature and the third hidden layer feature; wherein the decoding result of the current round of the autoregressive decoding is predicted by the output feature of the last decoding process in the current round of the autoregressive decoding; The execution result is obtained based on the decoding results of each round of the autoregressive decoding.

2. The method according to claim 1, characterized in that The selecting a target feed-forward layer from among a plurality of candidate feed-forward layers in the current decoding process based on the feature vector of the target word in the first hidden layer feature and determining a weight factor of the target feed-forward layer includes: Predicting based on the feature vector of the target word, obtaining a selection score for each of the plurality of candidate feed-forward layers; wherein the selection score of the candidate feed-forward layer represents the suitability of the candidate feed-forward layer for processing the first hidden layer feature; Based on the selection scores of the plurality of candidate feed-forward layers, the target feed-forward layer is selected from the plurality of candidate feed-forward layers and a weight factor of the target feed-forward layer is determined.

3. The method according to claim 2, characterized in that The selecting the target feed-forward layer from the candidate feed-forward layers based on the respective selection scores of the candidate feed-forward layers and determining the weight factor of the target feed-forward layer includes: Sorting the candidate feedforward layers based on their respective selection scores, and selecting a preset high-ranked candidate feedforward layer as the target feedforward layer; Based on the selection score of the target feed-forward layer, a weight factor of the target feed-forward layer is determined; wherein the candidate feed-forward layer that is not selected as the target feed-forward layer is configured with the weight factor being 0.

4. The method according to claim 1, wherein The step of weighting the second hidden layer feature after the first hidden layer feature is processed by the target feed-forward layer based on the weight factor of the target feed-forward layer to obtain the third hidden layer feature includes: Based on the weight factors of each target feed-forward layer, weighted summation is performed on the feature vectors of the current word unit in the first hidden layer features after being processed by the corresponding target feed-forward layer to obtain a weighted vector of the current word unit; The weighted vectors of each word in the first hidden layer feature are taken out from the continuous memory and combined in sequence to obtain the third hidden layer feature.

5. The method according to claim 1, wherein Before obtaining the output feature of the current decoding process based on the first hidden layer feature and the third hidden layer feature, the method further includes: Processing the first hidden layer features respectively based on a plurality of shared feedforward layers in the current decoding process to obtain fourth hidden layer features after the first hidden layer features are processed by each of the shared feedforward layers; Accumulating the fourth hidden layer features after being processed by each of the shared feedforward layers to obtain a fifth hidden layer feature; The obtaining, based on the first hidden layer feature and the third hidden layer feature, the output feature of the current decoding process includes: Based on the first hidden layer features, the third hidden layer features and the fifth hidden layer features, the output features of the current decoding process are obtained by fusing.

6. The method according to claim 1, wherein After performing attention calculation based on the output features of the previous decoding process of the current decoding process to obtain the first hidden layer features of the attention calculation, and before selecting a target feedforward layer from several candidate feedforward layers in the current decoding process based on the feature vector of the target word in the first hidden layer features and determining the weight factor of the target feedforward layer, the method further includes: Detecting whether the current decoding process includes the plurality of candidate feed-forward layers or only includes one feed-forward layer; The selecting a target feed-forward layer from among a plurality of candidate feed-forward layers in the current decoding process based on the feature vector of the target word in the first hidden layer feature and determining a weight factor of the target feed-forward layer includes: In response to the presence of the several candidate feed-forward layers in the current decoding process, the steps of selecting a target feed-forward layer from the several candidate feed-forward layers in the current decoding process and determining a weight factor of the target feed-forward layer based on the feature vector of the target word in the first hidden layer feature are performed.

7. The method according to claim 6, characterized in that The method further comprises: In response to the fact that there is only one feedforward layer in the current decoding process, the output features of the current decoding process are obtained by fusing the first hidden layer features and the sixth hidden layer features after the first hidden layer features are processed by the feedforward layer in the current decoding process.

8. The method according to claim 6, characterized in that At least one decoding process is selected from the plurality of decoding processes at a target interval, wherein the decoding process is provided with the plurality of candidate feed-forward layers.

9. The method according to any one of claims 1 to 8, characterized in that The target multimedia data includes at least one of image data, audio data, video data, and animation data; And / or, the target task includes: at least one of an image recognition task, a text recognition task, an audio recognition task, a video analysis task, and a picture expansion task; And / or, the target word is the last word in the first hidden layer feature; And / or, the execution result is obtained by a multimodal large model based on the target multimedia data and the prompt text, and the multimodal large model includes an encoder, an embedding layer and a decoder, the encoder is used to feature encode the target multimedia data, the embedding layer is used to text embed the prompt text, and the decoder is used to perform the autoregressive decoding.

10. A task execution device based on multimedia data, characterized in that: include: a fusion module configured to obtain a fused feature based on the encoding features of the target multimedia data and the embedded features of the prompt text; wherein the prompt text describes a target task to be performed on the target multimedia data in natural language, and the decoding results of the fused feature after multiple rounds of autoregressive decoding are used as the execution results of the target task on the target multimedia data, wherein each round of the autoregressive decoding includes multiple decoding processes executed sequentially; A decoding module configured to, during each round of the autoregressive decoding process: perform an attention calculation based on the output features of the previous decoding process of the current decoding process to obtain a first hidden layer feature of the attention calculation; select a target feedforward layer from a number of candidate feedforward layers in the current decoding process based on the feature vector of the target word in the first hidden layer feature and determine a weight factor of the target feedforward layer; weight a second hidden layer feature after the first hidden layer feature is processed by the target feedforward layer based on the weight factor of the target feedforward layer to obtain a third hidden layer feature; and obtain an output feature of the current decoding process based on the first hidden layer feature and the third hidden layer feature; wherein the decoding result of the current round of the autoregressive decoding is predicted by the output feature of the last decoding process in the current round of the autoregressive decoding; An acquisition module is used to obtain the execution result based on the decoding result of each round of the autoregressive decoding.

11. An electronic device, characterized in that: The method comprises at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor is used to execute the program instructions to implement the multimedia data-based task execution method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the task execution method based on multimedia data according to any one of claims 1 to 9.

Images