Training method of multi-task model, processing method and device of media data

Abstract

The application discloses a multi-task model training method, a media data processing method and device, and belongs to the technical field of computers. The method comprises the following steps: acquiring at least two target network models, any one of the target network models comprising a plurality of feature processing networks, the topological connection structures of the plurality of feature processing networks comprised by each target network model being the same, and any one of the target network models being used for performing one kind of task processing on target media data; merging the feature processing networks located at the same position in the topological connection structures of the at least two target network models, each feature processing network at one position being merged to obtain a first merged network; constructing a first task model based on each first merged network; and training the first task model to obtain a multi-task model. The application can quickly train the multi-task model, so that the multi-task model is used for task processing, and the task processing efficiency is improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a training method for a multi-task model, a method for processing media data, and an apparatus. Background Technology

[0002] A multi-task model is a neural network model that performs multiple tasks on media data. For example, it can be used to classify the mood of environmental descriptions in a script into positive, neutral, and negative emotions, and it can also be used to identify multiple emotions such as anger, fear, and fatigue in dialogue texts from movies and TV shows. Generally, a multi-task model needs to be trained first before it can be used for task processing. Therefore, how to train a multi-task model is a problem that urgently needs to be solved. Summary of the Invention

[0003] This application provides a method for training a multi-task model, a method for processing media data, and an apparatus that can quickly train a multi-task model for task processing, thereby improving task processing efficiency. The technical solution includes the following:

[0004] Firstly, a training method for a multi-task model is provided, the method comprising:

[0005] At least two target network models are obtained, each target network model includes multiple feature processing networks, and the topological connection structure formed by the multiple feature processing networks included in each target network model is the same. The target network model is used to perform a task processing on the target media data.

[0006] The feature processing networks located at the same position in the topological connection structure of the at least two target network models are merged, and the feature processing networks at one position are merged to obtain a first merged network.

[0007] Based on each of the first merged networks, construct the first task model;

[0008] The first task model is trained to obtain a multi-task model, which is used to perform at least one task processing on the target media data from the task processing corresponding to each target network model.

[0009] Secondly, a method for processing media data is provided, the method comprising:

[0010] Acquire target media data;

[0011] The target media data is processed by a multi-task model to obtain the results of each task. The multi-task model is trained according to the training method of the multi-task model shown in the first aspect.

[0012] Thirdly, a training device for a multi-task model is provided, the device comprising:

[0013] The acquisition module is used to acquire at least two target network models. Each target network model includes multiple feature processing networks. The topological connection structure formed by the multiple feature processing networks included in each target network model is the same. Each target network model is used to perform a task processing on the target media data.

[0014] The merging module is used to merge the feature processing networks located at the same position in the topological connection structure of the at least two target network models, and merge the feature processing networks at one position to obtain a first merged network.

[0015] The building module is used to construct the first task model based on each first merging network;

[0016] The training module is used to train the first task model to obtain a multi-task model, wherein the multi-task model is used to perform at least one task processing on the target media data from the task processing corresponding to each target network model.

[0017] In one possible implementation, each feature processing network includes a multi-layer processing sub-network, each processing sub-network includes network parameters for multiple channels, and the first merging network includes a multi-layer merging sub-network.

[0018] The merging module is used to select network parameters of a specified channel from the network parameters of each channel in any sub-processing layer of any feature processing network; and to construct a merged sub-network based on the network parameters of the specified channel corresponding to multiple sub-processing layers of each feature processing network at a given location.

[0019] In one possible implementation, the merging module is used to normalize the network parameters of each channel included in any layer of the processing sub-network to obtain normalized network parameters of each channel; based on the normalized network parameters of each channel, determine the importance index of each channel; and determine a specified channel from the channels whose importance index is not less than the index threshold, and obtain the network parameters of the specified channel.

[0020] In one possible implementation, the construction module is used to construct a first reconstruction network based on any first merging network and an initial control network for any first merging network, wherein the initial control network is used to control the output of the first reconstruction network; and to construct a first task model based on each first reconstruction network.

[0021] In one possible implementation, the training module is used to acquire sample media data and annotation results for each target network model. The sample media data corresponding to any target network model is used to train the target network model, and the annotation results of the target network model are used to annotate the processing results obtained by performing task processing on the sample media data. For the sample media data of the target network model, the first task model is used to perform corresponding task processing on the sample media data of the target network model to obtain the prediction result corresponding to the target network model. Based on the prediction results and annotation results corresponding to each target network model, the first task model is trained to obtain a multi-task model.

[0022] In one possible implementation, the various first reconstruction networks are arranged in multiple layers;

[0023] The training module is used to extract features from the sample media data of any target network model through the first reconstruction network of the first layer to obtain the media features output by the first reconstruction network of the first layer; for non-first reconstruction networks, it extracts features from the media features output by the first reconstruction network of the previous layer of the non-first reconstruction network to obtain the media features output by the non-first reconstruction network; and based on the media features output by the last first reconstruction network, it determines the prediction result corresponding to any target network model.

[0024] In one possible implementation, the first-layer first reconstruction network includes multiple first reconstruction networks, and the first merging network in the first reconstruction network includes an attention layer, a hidden layer and an output layer;

[0025] The training module is configured to: perform attention processing on sample media data of any target network model through the attention layer for any of the first reconstruction networks included in the first layer first reconstruction network, to obtain an attention processing result; perform feature crossing on the attention processing result through the hidden layer, to obtain a feature crossing result; adjust the feature crossing result based on the attention processing result through the initial adjustment network, to obtain an adjustment result; determine the output result based on the adjustment result through the output layer; and determine the media features output by the first layer first reconstruction network based on the output results of each of the first reconstruction networks included in the first layer first reconstruction network.

[0026] In one possible implementation, the initial control network includes a mapping network and an activation network, and the feature cross-result includes sub-results from multiple channels;

[0027] The training module is used to map the attention processing results through the mapping network to obtain the mapping results of each channel; to activate the mapping results of each channel through the activation network to obtain the activation results of each channel, wherein the activation results of the channels are used to characterize the activation degree of the channels; and to determine the regulation result based on the activation results of each channel and the sub-results of the multiple channels.

[0028] In one possible implementation, the training module is used to train the first task model according to the first loss and the second loss corresponding to each target network model to obtain a multi-task model.

[0029] The first loss corresponding to any target network model is determined based on the activation results of each channel corresponding to each first reconstruction network.

[0030] The second loss corresponding to any target network model is determined based on the prediction results and annotation results of the target network model.

[0031] In one possible implementation, the training module is used to train each initial control network in the first task model to obtain each first control network; and to alternately train each first merging network and each first control network to obtain a multi-task model.

[0032] Fourthly, a media data processing apparatus is provided, the apparatus comprising:

[0033] The acquisition module is used to acquire target media data;

[0034] The task processing module is used to perform at least one task processing on the target media data through a multi-task model to obtain the results of each task processing. The multi-task model is trained according to the training method of the multi-task model shown in the first aspect.

[0035] In one possible implementation, the multi-task model includes a multi-layer target reconstruction network;

[0036] The task processing module is used to extract features from the target media data through the first-layer target reconstruction network to obtain the media features output by the first-layer target reconstruction network; for non-first-layer target reconstruction networks, it extracts features from the media features output by the target reconstruction network above the non-first-layer target reconstruction network to obtain the media features output by the non-first-layer target reconstruction network; and determines the processing results of each task based on the media features output by the last-layer target reconstruction network.

[0037] Fifthly, an electronic device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, the at least one computer program being loaded and executed by the processor to enable the electronic device to implement the training method of the multi-task model described in the first aspect or the media data processing method described in the second aspect.

[0038] In a sixth aspect, a computer-readable storage medium is also provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to enable an electronic device to implement the training method of the multi-task model described in the first aspect or the media data processing method described in the second aspect.

[0039] In a seventh aspect, a computer program is also provided, wherein the computer program is at least one, and the at least one computer program is loaded and executed by a processor to enable an electronic device to implement the training method of the multi-task model shown in the first aspect or the media data processing method shown in the second aspect.

[0040] Eighthly, a computer program product is also provided, wherein at least one computer program is stored therein, the at least one computer program being loaded and executed by a processor to enable an electronic device to implement the training method of the multi-task model shown in the first aspect or the media data processing method shown in the second aspect.

[0041] The technical solution provided in this application brings at least the following beneficial effects:

[0042] In the technical solution provided in this application, each target network model includes multiple feature processing networks, and the topological connection structure formed by the multiple feature processing networks in each target network model is the same. By merging the feature processing networks located at the same position in the topological connection structure, each first merged network is obtained, which realizes the transfer of network parameters of the feature processing networks to the first merged network, which is beneficial to accelerating the training efficiency of the first merged network. A first task model is constructed through each first merged network, and a multi-task model is obtained by training the first task model, which realizes the rapid training of the multi-task model for task processing, thereby improving the task processing efficiency. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram illustrating the implementation environment of a multi-task model training method or a media data processing method provided in the embodiments of this application;

[0045] Figure 2 This is a flowchart of a training method for a multi-task model provided in an embodiment of this application;

[0046] Figure 3 This is a schematic diagram of a BERT fine-tuning method provided in an embodiment of this application;

[0047] Figure 4 This is a schematic diagram illustrating the construction of a first reconstructed network provided in an embodiment of this application;

[0048] Figure 5 This is a schematic diagram illustrating the construction of a first task model provided in an embodiment of this application;

[0049] Figure 6 This is a schematic diagram illustrating a text processing task provided in an embodiment of this application;

[0050] Figure 7 This is a flowchart of a media data processing method provided in an embodiment of this application;

[0051] Figure 8 This is a schematic diagram illustrating a task processing method using a multi-task model, as provided in an embodiment of this application.

[0052] Figure 9 This is a flowchart illustrating the training process of a multi-task model provided in an embodiment of this application;

[0053] Figure 10 This is a schematic diagram of a first-stage training provided in an embodiment of this application;

[0054] Figure 11 This is a flowchart of a second-stage training process provided in an embodiment of this application;

[0055] Figure 12 This is a schematic diagram of the structure of a training device for a multi-task model provided in an embodiment of this application;

[0056] Figure 13 This is a schematic diagram of the structure of a media data processing device provided in an embodiment of this application;

[0057] Figure 14 This is a schematic diagram of the structure of a terminal device provided in an embodiment of this application;

[0058] Figure 15 This is a schematic diagram of the structure of a server provided in an embodiment of this application. Detailed Implementation

[0059] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0060] It should be noted that the terms "first," "second," etc., used in this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0061] In the field of artificial intelligence technology, a multi-task model can be trained first, and then used to process media data for various tasks. However, how to train a multi-task model is a problem that urgently needs to be solved. Based on this, embodiments of this application provide a method for training a multi-task model or a method for processing media data, which can quickly train a multi-task model with high accuracy, and use the multi-task model to process target media data to obtain highly accurate task processing results.

[0062] like Figure 1 As shown, Figure 1This is a schematic diagram illustrating the implementation environment of a multi-task model training method or a media data processing method provided in this application embodiment. The implementation environment includes a terminal device 101 and a server 102. The multi-task model training method or media data processing method in this application embodiment can be executed by the terminal device 101, by the server 102, or jointly by the terminal device 101 and the server 102.

[0063] Terminal device 101 can be a smartphone, game console, desktop computer, tablet computer, laptop computer, smart TV, smart in-vehicle device, smart voice interaction device, smart home appliance, etc. Server 102 can be a single server, a server cluster consisting of multiple servers, or any of the following: cloud computing platform and virtualization center. This application embodiment does not limit this. Server 102 can be connected to terminal device 101 through a communication network, which can be a wired network or a wireless network. Server 102 can have functions such as data processing, data storage, and data transmission and reception. This application embodiment does not limit this. The number of terminal devices 101 and servers 102 is not limited; there can be one or more.

[0064] In this embodiment, the terminal device 101 or server 102 trains a multi-task model based on the training method of the multi-task model in this embodiment. In one possible implementation, the terminal device 101 or server 102 acquires target media data and, according to the media data processing method of this embodiment, performs multiple task processing on the target media data using the multi-task model to obtain multiple task processing results. In another possible implementation, the terminal device 101 acquires the target media data and sends the target media data to the server 102 via a communication network. The server 102, according to the media data processing method of this embodiment, performs multiple task processing on the target media data using the multi-task model to obtain multiple task processing results and sends the multiple task processing results to the terminal device 101 via a communication network.

[0065] The process by which terminal device 101 or server 102 trains the multi-task model is as follows: Terminal device 101 or server 102 can obtain target network models 1 to M, where M is a positive integer greater than 1. The target network model m (m is a positive integer taking any value from 1 to M) includes multiple feature processing networks and prediction networks m, where the i-th (i is a positive integer) feature processing network can be represented as bmi. For example... Figure 1 In the target network model 1, there are 12 feature processing networks, designated b11 to b112; target network model M also includes 12 feature processing networks, designated bM1 to bM12. Each target network model is used to perform a task on media data, such as... Figure 1 In this context, target network model 1 is used for task processing 1 of media data, and target network model M is used for task processing M of media data.

[0066] For the i-th feature processing network, a first reconstruction network Ebi in the multi-task model can be constructed based on the i-th feature processing network in the target network models 1 to M. For example, Eb1 is constructed based on b11 to bM1, Eb2 is constructed based on b12 to bM2, and so on. Following this method, each first reconstruction network is constructed. The first reconstruction networks Eb1 to Eb12 are then concatenated with prediction networks 1 to M, and these prediction networks 1 to M are connected in parallel, thus constructing the first task model. By training the first task model, the multi-task model is obtained. The multi-task model is used to perform task processing 1 to M on the media data. For example, the target media data is input into the multi-task model, and the multi-task model performs task processing 1 to M on the target media data, obtaining task processing results 1 to M respectively.

[0067] The various optional implementations of this application's embodiments can be based on artificial intelligence (AI) technology. AI is the theory, methods, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can react in a way similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0068] Artificial intelligence (AI) is a comprehensive discipline encompassing a wide range of fields, including both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, pre-trained model technology, operating / interactive systems, and mechatronics. Among these, pre-trained models, also known as large-scale models or foundational models, can be widely applied to downstream tasks across various AI fields after fine-tuning. AI software technologies primarily include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0069] like Figure 2 As shown, Figure 2 This application provides a training method for a multi-task model, which can be applied to the above-described implementation environment. The method in this application is executed by an electronic device, including a terminal device 101 or a server 102, and includes the following steps.

[0070] Step 201: Obtain at least two target network models. Each target network model includes multiple feature processing networks. The topological connection structure formed by the multiple feature processing networks in each target network model is the same. Each target network model is used to perform a task processing on the target media data.

[0071] This application does not limit the method by which an electronic device acquires any target network model. For example, the electronic device can train the target network model, or it can read a pre-trained target network model from another device, and so on. In this application embodiment, regardless of the method by which the electronic device acquires the target network model, at least two target network models are acquired, and each target network model is used to perform a task processing on the target media data. The target media data includes, but is not limited to, at least one of text, images, audio, and video. The task processing includes at least one of emotion recognition, classification, translation, question answering, and target object detection.

[0072] Understandably, different target network models can correspond to the same or different target media data. For example, one target network model might correspond to dialogue text, while another might correspond to environmental description text. Different target network models can also correspond to the same or different task processing. For example, one target network model might be used for classification, while another might be used for emotion recognition.

[0073] In an exemplary embodiment, multiple target network models are required to perform corresponding task processing on various types of target media data involved in a certain scene. For example, a script includes dialogue text and environmental description text. One target network model can be used to perform emotion recognition on the dialogue text, including 13 emotions such as belief, resentment, joy, fear, expectation, worry, doubt, love, hate, anger, fatigue, indifference, and contempt. Another target network model can be used to perform emotion recognition on the environmental description text, including positive, neutral, and negative emotions.

[0074] Furthermore, the target network model in this application embodiment includes multiple feature processing networks, which form a topological connection structure. This application embodiment does not limit the type of topological connection structure. For example, every two consecutive feature processing networks are connected end-to-end to form a serial topological connection structure. Alternatively, these feature processing networks form multiple layers, each layer including several feature processing networks. Any feature processing network in each layer is connected to each feature processing network in the next layer, thereby forming a hybrid topological connection structure.

[0075] The structure of the feature processing network is not limited in this application. Exemplarily, the feature processing network includes at least one of the following network layers: linear layers, convolutional layers, activation layers, normalization layers, attention layers, and dropout layers. Optionally, the feature processing network is BERT (Bidirectional Encoder Representations from Transformers) or BERT-WWM (Whole Word Masking).

[0076] In one possible implementation, the target network model includes a base model module and a task head module. The base model module comprises cascaded multi-layer BERT, for example, a 12-layer BERT. BERT is an open-source language model that can be pre-trained using large-scale language data, and then fine-tuned for downstream tasks. The fine-tuning process is as follows: Figure 3 As shown.

[0077] BERT can be applied to a variety of downstream tasks, such as MNLI (Multi-Genre Natural Language Inference), NER (Named Entity Recognition), and SQuAD (Stanford Question Answering Dataset). The methods for fine-tuning BERT differ slightly across these downstream tasks. The following section uses SQuAD as an example to illustrate the BERT fine-tuning process.

[0078] like Figure 3 As shown, the SQuAD task typically requires a text pair consisting of Text 1 and Text 2. Text 1 is the descriptive text for the question-and-answer task; for example, Text 1 is "Please give the speaker's emotion based on the following sentence." Text 1 includes characters 1 to N, which can be mapped to corresponding character vectors (i.e., embeddings) using a dictionary, resulting in character vectors E1 to E2. N Text 2 is the statement text to be judged, used to provide the answer to Text 1. For example, Text 2 is "I gritted my teeth, my lips trembled, my hands gripped the bedpost tightly, and I was covered in cold sweat." Similar to Text 1, Text 2 includes characters 1 to M. A dictionary can be used to map characters 1 to M to their corresponding character vectors, resulting in character vectors E1′ to E′. M Where M and N are both positive integers greater than 1, and M and N may be equal or unequal.

[0079] In addition, the text 1 is preceded by a start character [CLS], and the character vector E corresponding to the start character [CLS] can be determined using a dictionary. [CLS] The text between text 1 and text 2 includes a separator character [SEP], which can be determined using a dictionary to identify the character vector E corresponding to the separator character [SEP]. [SEP] The delimiter character is used to separate text 1 and text 2, allowing BERT to distinguish between the description text of the question-answering task and the text of the statement to be decided.

[0080] BERT comprises a multi-layer Transformer structure. In practical applications, BERT can include a 12-layer Transformer structure, which includes, but is not limited to, hidden layers and self-attention (SA) layers. Optionally, the hidden layers have a dimension of 768, and the SA layers are multi-head self-attention layers with 12 heads. The total number of parameters in BERT is approximately 110 million, and the number of parameters does not change during fine-tuning. To concisely describe the BERT structure, as follows... Figure 3 As shown, only two Transformer structures are shown, where the circle indicated by label 301 represents the Transformer structure.

[0081] By using character vector E [CLS] Character vectors E1 to E N Character vector E [SEP] Character vectors E1′ to E′ M The concatenation is performed to obtain a concatenated vector, which is then input into BERT. Feature extraction is performed on the concatenated vector through each Transformer layer, resulting in the target vector output by the final Transformer layer. The target vector includes the feature vector C corresponding to the start character [CLS]. Feature vector C can represent the semantics of the answer to text 1 determined based on text 2. For example, in an emotion recognition task, feature vector C can represent the semantics of the emotion implied in text 2. The target vector also includes feature vectors T1 to T2 corresponding to characters 1 to N, respectively. N The feature vector T corresponding to the separator character [SEP] [SEP] Characters 1 to M correspond to feature vectors T1′ to T′ respectively. M .

[0082] As mentioned above, the target network model can include multiple layers of BERT. In a non-last BERT layer, the target vector output by the last Transformer structure is the input to the first Transformer structure in the next BERT layer. In the last BERT layer, the target vector output by the last Transformer structure can be used to determine the task processing result of downstream tasks. In this example, a feature processing network can be either a BERT or a Transformer structure.

[0083] Step 202: Merge the feature processing networks located at the same position in the topology of at least two target network models, and merge the feature processing networks at one position to obtain a first merged network.

[0084] In this embodiment, a target network model includes multiple feature processing networks forming a certain topological connection structure, with each feature processing network occupying a different position within that topological connection structure. Since there are at least two target network models, and the multiple feature processing networks included in each target network model correspond to the same topological connection structure, each feature processing network located at the same position in the topological connection structure belongs to its respective target network model. For example, each feature processing network located at the same position in a serial topological connection structure is equivalent to the i-th (i is a positive integer) feature processing network in each target network model; each feature processing network located at the same position in a hybrid topological connection structure is equivalent to the j-th (j is a positive integer) feature processing network in the i-th (i is a positive integer) layer of each target network model.

[0085] For feature processing networks located at the same position in the topology, the electronic device can merge these feature processing networks into a first merged network, and the merging method is not limited here. For example, the network structures of the feature processing networks are identical; that is, any feature processing network includes N layers of cascaded processing sub-networks. The i-th layer (where i takes any positive integer from 1 to N) of the processing sub-networks included in each feature processing network is concatenated to obtain a merged sub-network. The number of network parameters in the merged sub-network is equal to the number of feature processing networks multiplied by the number of network parameters in the i-th layer. The N merged sub-networks form the first merged network.

[0086] In one possible implementation, any feature processing network includes a multi-layer processing sub-network, each processing sub-network includes network parameters for multiple channels, and a first merging network includes a multi-layer merging sub-network. Step 202 includes, but is not limited to, steps 2021 to 2022 (not shown in the figure).

[0087] In this embodiment, the feature processing network includes at least one network layer selected from linear layers, convolutional layers, activation layers, normalization layers, attention layers, and dropout layers, with each network layer constituting a processing sub-network. Optionally, the target network model includes BERT, which comprises a 12-layer Transformer structure. Each Transformer structure is a feature processing network, and the network layers included in the Transformer structure are shown in Table 1 below. Here, "-" indicates no network parameters to be learned.

[0088] Table 1

[0089]

[0090]

[0091] As shown in Table 1, the Transformer structure includes Attention and FFN (Feed-Forward Network). Attention can be Self Attention (SA) or Cross Attention (CA). The feed-forward network includes Intermediate (hidden) and Output layers. Attention consists of 12 Headers used to learn basic information about the input. The input Attention feature dimension is 768. The input FFN feature dimension is also 768. During feature processing using FFN, the feature dimension is expanded from 768 to 3072. Feature crossing is performed on the 3072-dimensional features, and then the resulting features are compressed back to 768 dimensions. In this example, a feature processing network is a Transformer structure, comprising multiple sub-networks: an attention layer, hidden layers, and an output layer.

[0092] Step 2021: For any feature processing network, which includes any layer of processing sub-network, select the network parameters of a specified channel from the network parameters of each channel included in the processing sub-network.

[0093] In this embodiment, any processing sub-network layer includes network parameters for multiple channels. A channel is a processing unit in the processing sub-network, used to perform operations on the input features to obtain the results. For example, when the processing sub-network is an attention layer, the attention layer includes 12 heads, each head is used to perform attention processing on the features based on the attention mechanism. Therefore, a head is a channel of the attention layer, and the network parameters of the head are the network parameters of the channel. That is, the attention layer includes network parameters for 12 channels. As another example, when the processing sub-network is a hidden layer or output layer in an FFN, since the FFN expands the feature dimension from 768 dimensions to 3072 dimensions, for each feature dimension, feature cross is performed between that feature dimension and at least one other feature dimension. Therefore, the FFN can be considered to include 3072 processing units, each processing unit being a channel. That is, both the hidden layer and the output layer include network parameters for 3072 channels.

[0094] The electronic device can select network parameters for a specified channel from the network parameters of each channel included in any layer of the processing sub-network. The selection method is not limited in this embodiment. For example, the electronic device can select the channel selected by the manual selection operation for each channel as the specified channel and obtain the network parameters of the specified channel. Alternatively, the electronic device can select the network parameters of the specified channel from the network parameters of each channel according to pre-configured rules, where the specified channel can be at least one.

[0095] For example, the processing sub-network layer may include an attention layer, a hidden layer, or an output layer. As shown in Table 1 above, the attention layer, hidden layer, and output layer are all linear layers; therefore, the network parameters for each channel include weights. Taking any processing sub-network layer as a hidden layer as an example, the electronic device selects the network parameters for a specified channel according to the following rules: the 3072 channels in the hidden layer are divided into groups of four, resulting in 768 groups of channels. The first channel in each group is the specified channel. Based on this, the electronic device can obtain the weights of the 768 specified channels.

[0096] In an exemplary embodiment, step 2021 includes: normalizing the network parameters of each channel in any layer of the processing sub-network to obtain normalized network parameters for each channel; determining the importance index of each channel based on the normalized network parameters of each channel; and determining a specified channel from the channels whose importance index is not less than the index threshold, and obtaining the network parameters of the specified channel.

[0097] Generally, only a portion (or a small portion) of the network parameters in the processing sub-network are used to provide task-related information, another portion is used to provide some generalized data prediction information, and the remaining portion has a relatively small role. By determining the importance index of each channel, selecting designated channels with importance indices not less than the index threshold, and obtaining the network parameters of the designated channels, it is possible to obtain the important parameters in the original network and filter out redundant parameters in the original network.

[0098] For any given channel, the network parameters of that channel can be normalized using a normalization function. Alternatively, when the network parameters of a channel are N-dimensional vectors (e.g., one-dimensional vectors), the normalized network parameters of that channel can be obtained by dividing the magnitude of the network parameters of that channel by the sum of the magnitudes of the network parameters of all other channels. Optionally, the normalized network parameters of any channel are greater than 0 and less than 1.

[0099] You can follow Calculate the total entropy H(X) of any layer processing subnetwork. Where p i This characterizes the normalized network parameters of the i-th channel in the processing sub-network of this layer. The information entropy of the i-th channel is used to characterize the channel's information content. The higher the information entropy of a channel, the greater its information content and the higher its importance. Therefore, the information entropy of the i-th channel can be used as an indicator of its importance. In practical applications, other methods for determining importance indicators can also be used, which will not be elaborated upon here.

[0100] For the i-th channel, if the importance index of the i-th channel is not less than the index threshold, then the i-th channel is designated as the specified channel, and the network parameters of the specified channel are obtained. This application does not limit the index threshold; for example, the index threshold is a value set based on human experience, or the index threshold is an important index among the importance indices of each channel.

[0101] Optionally, assuming any processing sub-network includes K (K is a positive integer) channels of network parameters, the electronic device can sort the K channels according to their importance index from largest to smallest, and select the network parameters of the top k (k takes any value from 1 to K) channels as the network parameters of the specified channel. Optionally, assuming there are m (m is a positive integer) target network models, then k = K / m. For example, if there are 2 target network models, and any processing sub-network is a hidden layer with 3072 channel weights, the electronic device can select the weights of the top 3072 / 2 = 1536 channels as the weights of the specified channel.

[0102] Step 2022: For any processing sub-network included in each feature processing network of a location, construct a merged sub-network based on the network parameters of the specified channel corresponding to any processing sub-network included in each feature processing network of a location.

[0103] In this way, multi-layered merging sub-networks can be determined, thus obtaining the first merging network. By constructing merging sub-networks using network parameters of specified channels, the first merging network is obtained. This allows important parameters from the original network to be transferred to the first merging network while filtering out redundant parameters from the original network. This ensures the accuracy of the first merging network, reduces the amount of redundant parameters, and improves the computational efficiency of the first merging network.

[0104] As mentioned above, for any layer of processing subnetworks included in any feature processing network, the network parameters of a specified channel can be selected. Based on this, an initial subnetwork of the same type as the layer processing subnetwork can be constructed, and the network parameters of each channel in the initial subnetwork can be initialized to the network parameters of the specified channels of the layer processing subnetworks included in each feature processing network located at the same position in the topology connection structure, thus obtaining a merged subnetwork.

[0105] Let's take two tasks as an example. These two tasks are Task 1 and Task 2, each corresponding to a target network model, which includes a feature processing network. In this example, the feature processing network is as follows: Figure 4 The Transformer structure shown includes an attention layer, a hidden layer, and an output layer.

[0106] We can select 6 channels of network parameters from the attention layer corresponding to Task 1, and 6 channels of network parameters from the attention layer corresponding to Task 2. Then, we initialize the 12 channels of network parameters in the attention layer of the expert Transformer structure to the selected 12 channels of network parameters. At this point, the attention layer in the expert Transformer structure includes network parameters from the attention layer corresponding to Task 1 (such as...). Figure 4 The black areas in the attention layer of the expert Transformer structure shown) and the network parameters from the attention layer corresponding to Task 2 (such as...) Figure 4 (The white area in the attention layer of the expert Transformer structure is shown). Following a similar process to the attention layer, the hidden layer and output layer of the expert Transformer structure are obtained, thus yielding the first merging network, which includes the attention layer, hidden layer, and output layer of the expert Transformer structure.

[0107] Step 203: Construct the first task model based on each first merging network.

[0108] As mentioned above, feature processing networks located at the same position in the topological connection structure belong to different target network models. Feature processing networks at the same position in each target network model can be merged to form a first merged network. In other words, one position in the topological connection structure corresponds to one first merged network. Based on this, there are multiple first merged networks, and these multiple first merged networks can form the topological connection structure. Therefore, a first task model can be constructed based on each of these first merged networks.

[0109] In an exemplary embodiment, step 203 includes steps 2031 to 2032 (not shown in the figure).

[0110] Step 2031: For any first merging network, construct a first reconstruction network based on any first merging network and the initial control network. The initial control network is used to control the output of the first reconstruction network.

[0111] In this embodiment, a first reconstruction network is constructed based on a first merging network and an initial control network. This not only transfers the network parameters of the feature processing network to the first reconstruction network, accelerating the training efficiency of the first reconstruction network, but also controls the output of the first reconstruction network through the initial control network. This allows the first reconstruction network to adaptively capture the information that different tasks are concerned with, avoid conflicts between different tasks, and improve the accuracy of the output results of the first reconstruction network, thereby improving the accuracy of the model's processing results.

[0112] After constructing the first merging network, the electronic device can connect the first merging network and the initial control network to construct the first reconstructed network, thereby enabling the control of the output of the first reconstructed network through the initial control network. This application embodiment does not limit the network structure of the initial control network. For example, the initial control network includes at least one of linear layers, fully connected layers, activation layers, and convolutional layers. Optionally, a statistical distribution function is used to initialize the network parameters of the initial control network. This application embodiment does not limit the type of statistical distribution function; for example, a Gaussian normal distribution with a mean of 0 and a variance of 1 is used to initialize the network parameters of the initial control network.

[0113] Optionally, the initial control network includes a mapping network and an activation network. The mapping network is used to perform linear or nonlinear mapping on the input features, and the activation network is used to activate the input features. The mapping network includes at least one of linear layers or fully connected layers.

[0114] like Figure 4As shown, the expert Transformer structure is the first reconstruction network in this embodiment. The first reconstruction network includes a first merging network, which includes an attention layer, a hidden layer, and an output layer. The first reconstruction network also includes an initial control network, which includes an expert module and a gating module. The expert module can be the mapping network mentioned above, and the gating module can be the activation network mentioned above. The connections of each network layer in the first reconstruction network are as follows: Figure 4 As shown, it will not be elaborated further here.

[0115] Step 2032: Construct the first task model based on each first reconstruction network.

[0116] In this embodiment, the first reconstruction network has high training efficiency and high accuracy in its output. Constructing a first task model using various first reconstruction networks facilitates rapid and accurate training of a multi-task model, enabling task processing and improving both efficiency and accuracy.

[0117] As mentioned above, for each feature processing network at the same location in the topological connection structure, these feature processing networks belong to their respective target network models, and a first reconstruction network can be constructed based on the initial control network and these feature processing networks. In other words, one location in the topological connection structure corresponds to one first reconstruction network. Based on this, the first reconstruction networks corresponding to each location in the topological connection structure can form the topological connection structure itself, thereby enabling the construction of a first task model based on multiple first reconstruction networks that form this topological connection structure.

[0118] Optionally, in addition to multiple feature processing networks, the target network model also includes a prediction network. The prediction network maps the features output by the multiple feature processing networks to the prediction results of the task processing corresponding to the target network model. The type of prediction network is not limited here. For example, the prediction network includes at least one of fully connected layers, convolutional layers, sigmoid activation layers, and softmax activation layers.

[0119] The prediction networks of each target network model can be concatenated after multiple first reconstruction networks to obtain the first task model. The prediction networks of each target network model are connected in parallel.

[0120] like Figure 5 As shown, the first task model includes a first reconstruction network Eb1 to Eb12 connected in series. One of the first reconstruction networks is... Figure 4The expert Transformer structure is shown. After the first reconstruction networks Eb1 to Eb12, prediction network 1 and prediction network 2 are connected in series respectively. Prediction network 1 is the prediction network corresponding to task 1, and prediction network 2 is the prediction network corresponding to task 2. Task 1 and task 2 are two tasks.

[0121] Step 204: Train the first task model to obtain a multi-task model. The multi-task model is used to process the target media data using at least one of the tasks corresponding to each target network model.

[0122] In this embodiment, the first task model can be trained once to obtain a trained first task model. If the trained first task model meets the training termination condition, it is used as a multi-task model. If the trained first task model does not meet the training termination condition, it is used as the first task model for the next training iteration, and the first task model is trained again according to steps 201 to 204 to obtain a trained first task model. The multi-task model is then determined based on the trained first task model.

[0123] This application does not limit the conditions under which the first task model after training meets the training termination criteria. For example, the conditions under which the first task model after training meets the training termination criteria include: the number of training iterations of the first task model after training reaches a set number, or the error of the first task model after training is within a set error range, etc. Optionally, the set number of iterations and the set error range are values ​​set based on human experience.

[0124] In practical applications, there are multiple ways to train the first task model. Below, we will introduce two possible implementation methods from the perspectives of implementation method A and implementation method B.

[0125] In implementation method A, step 204 includes steps A1 to A3 (not shown in the figure).

[0126] Step A1: Obtain sample media data and annotation results for each target network model. The sample media data of any target network model is used to train any target network model, and the annotation results corresponding to any target network model are used to annotate the processing results obtained by processing the sample media data.

[0127] For any given target network model, it can be trained using multiple sample media data and the annotation results of each sample media data. The training method will not be elaborated here. Electronic devices can acquire these sample media data and annotation results. This application does not limit the method of acquiring sample media data. For example, electronic devices can acquire sample media data input by the user, or electronic devices can acquire sample media data via the Internet, or electronic devices are equipped with acquisition devices such as cameras and recorders to acquire sample media data.

[0128] The annotation results of the sample media data are obtained through annotation and are used to characterize the processing results of the target network model corresponding to the sample media data. It is understandable that the annotation results will differ depending on the task processing. For example, in an emotion recognition task, multiple emotion labels can be preset. The annotation results of the sample media data include the annotation values ​​of multiple emotion labels, each used to characterize whether the sample media data belongs to a different emotion label. Optionally, if the annotation value of an emotion label is 1, the annotation value indicates that the sample media data belongs to an emotion label; if the annotation value of an emotion label is 0, the annotation value indicates that the sample media data does not belong to an emotion label. Based on the annotation results of the sample media data, it can be determined that the sample media data belongs to at least one emotion label among multiple emotion labels. Examples of three emotion recognition tasks are introduced below.

[0129] In Example 1, the emotion recognition task involves multi-emotion identification of dialogue lines. In this case, the sample media data is dialogue lines, and the pre-set emotion labels for the dialogue lines include 13 emotion labels: belief, resentment, joy, fear, expectation, worry, doubt, love, hate, anger, fatigue, indifference, and contempt. Example 1: The sample media data is "I clenched my teeth, my lips trembled, my hands gripped the bed leg tightly, and I was covered in cold sweat." The emotion label for this sample media data is "fear," so the labeling result for this sample media data is 0001000000000, with the label value for "fear" being 1, and the label values ​​for other emotion labels being 0. Example 2: The sample media data is "A: Yes, I just believe her! Don't give me any high-sounding reasons. In the end, aren't you just jealous of B?" The emotional labels of this sample media data are "belief" and "resentment". The labeling result of this sample media data is 11000000000000, with the labeling values ​​of "belief" and "resentment" both being 1, and the labeling values ​​of other emotional labels being 0.

[0130] In Example 2, the emotion recognition task includes emotion recognition of environmental description text. In this case, the sample media data is environmental description text, and the pre-set emotion tags for the environmental description text include: fighting spirit, enthusiasm, indifference, horror, joy, tension, fear, and other emotions. For the environmental description text "The building became rocks on the beach, and huge waves crashed in, with yellow muddy water, scattered buildings, and floating garbage in the waves...", the emotion tag for this environmental description text is "fear". Therefore, the annotation result for this sample media data is 00000010, the annotation value for "fear" is 1, and the annotation values ​​for other emotion tags are all 0.

[0131] In Example 3, the emotion recognition task includes emotion recognition of environmental description text. In this case, the sample media data is environmental description text, and the pre-set emotion labels for the environmental description text include: positive (e.g., full of fighting spirit, enthusiasm, joy, etc. are positive), neutral, and negative (e.g., indifference, horror, tension, fear, etc. are negative). For the environmental description text "The building became rocks on the beach, and wave after wave crashed in, with yellow muddy water, scattered buildings, and floating garbage in the waves...", the emotion label for this environmental description text is "negative". Therefore, the labeling result for this sample media data is 001, the labeling value for "negative" is 1, and the labeling values ​​for "positive" and "neutral" are both 0.

[0132] Step A2: For any target network model's sample media data, the first task model is used to perform corresponding task processing on the sample media data of any target network model to obtain the prediction result corresponding to any target network model.

[0133] In this embodiment, the first task model includes multiple first reconstruction networks and prediction networks for each target network model. The multiple first reconstruction networks extract features from the sample media data of any target network model to obtain sample media features, which characterize the semantics of the sample media data. The prediction networks of the target network models map the sample media features to prediction results. These prediction results are determined by the first task model and characterize the processing result obtained after performing the task corresponding to the target network model on the sample media data of that target network model.

[0134] Understandably, different tasks correspond to different prediction results. For example, in an emotion recognition task involving multiple emotions in dialogue, which corresponds to 13 emotion labels, the prediction result includes 13 probabilities, each representing the probability that the sample media data belongs to a different emotion label. In an emotion recognition task involving emotion recognition in environmental description text, which corresponds to positive, neutral, and negative emotions, the prediction result includes three probabilities, representing the probability that the sample media data belongs to positive, neutral, and negative emotions, respectively.

[0135] As mentioned above, multiple first reconstruction networks form a topological connection structure, and different topological connection structures correspond to different feature extraction methods. In an exemplary embodiment, the first reconstruction networks are arranged in multiple layers, and each layer of the first reconstruction network includes at least one first reconstruction network. In this case, step A2 includes steps A21 to A23 (not shown in the figure).

[0136] Step A21: For the first layer first reconstruction network, feature extraction is performed on the sample media data of any target network model through the first layer first reconstruction network to obtain the media features output by the first layer first reconstruction network.

[0137] In this embodiment, the sample media data of any target network model includes multiple media data segments. For example, when the sample media data is text, the media data segments are words or characters; when the sample media data is audio, the media data segments are audio frames; when the sample media data is an image, the media data segments are regions in the image; when the sample media data is video, the media data segments are frame images. For ease of description, the following explanation uses the example of sample media data being text and media data segments being words.

[0138] The sample media data includes multiple words. Electronic devices can use a dictionary (e.g., vocab.txt) to map each word to a corresponding dictionary ID (Identity Document, also known as a word vector). An embedding vector is then determined based on each word vector. For example, the embedding vector can be obtained by concatenating the word vectors, or by starting with the character vector E corresponding to the beginning character [CLS]. [CLS] Then, the word vectors are concatenated sequentially to obtain the embedding vector.

[0139] After obtaining the embedding vector, it can be input into the first layer of the first reconstruction network. The first layer of the first reconstruction network extracts features from the embedding vector, resulting in the media features output by the first layer of the first reconstruction network. It is understandable that different structures of the first layer of the first reconstruction network will result in different feature extraction methods.

[0140] In one possible implementation, the first-layer first reconstruction network includes multiple first reconstruction networks, and the first merging network in the first reconstruction network includes an attention layer, a hidden layer, and an output layer. Step A21 includes steps A211 to A212 (not shown in the figure).

[0141] Step A211: For any of the first reconstruction networks included in the first reconstruction network of the first layer, attention processing is performed on the sample media data of any target network model through the attention layer to obtain the attention processing result; feature crossing is performed on the attention processing result through the hidden layer to obtain the feature crossing result; the feature crossing result is controlled based on the attention processing result through the initial control network to obtain the control result; and the output result is determined based on the control result through the output layer.

[0142] In this embodiment of the application, any first reconstructed network includes an initial control network, an attention layer, a hidden layer, and an output layer. The initial control network is connected to the attention layer, the hidden layer, and the output layer, and the attention layer is connected to the hidden layer.

[0143] First, an embedding vector is determined based on sample media data from any target network model. This embedding vector is then input into an attention layer, which performs attention processing on the embedding vector using an attention mechanism to obtain the attention result. The attention layer can include, but is not limited to, any of the following: self-attention layer, cross-attention layer, multi-head attention layer, or masked attention layer. Different types of attention layers perform attention processing differently, which will not be elaborated upon here.

[0144] In one possible implementation, the embedding vector is weighted three times by an attention layer to obtain a query vector, a key vector, and a value vector. The weights of the value vectors are determined by the query vector and the key vector, and the value vectors are then weighted using these weights to obtain the attention processing result, thereby achieving attention processing on the embedding vectors.

[0145] As mentioned above, the attention layer includes network parameters for multiple channels. For each channel, attention processing is performed on the embedding vector using the channel's network parameters to obtain the channel's first result. In other words, the attention processing result includes the first results for multiple channels. This attention processing result is then input into the hidden layer, which expands the first results for multiple channels to increase the number of channels, resulting in multiple expanded first results. For example, the hidden layer expands the first results for 768 channels into first results for 3072 channels. The hidden layer also includes network parameters for multiple channels. For each channel, feature crossing is performed on the first results for at least two expanded channels using the channel's network parameters to obtain the feature cross result. Feature crossing, also called feature combination, is a multiplication operation. By multiplying features, a non-linear transformation of the features is achieved, increasing non-linearity.

[0146] The feature crossing result includes the first result after feature crossing of multiple expanded channels. For ease of description, the first result after feature crossing of expanded channels is referred to as the channel's sub-result. The attention processing result and the feature crossing result are input into the initial control network. The initial control network controls the feature crossing result based on the attention processing result to obtain the control result. The control method is not limited here. Optionally, based on the attention processing result, the initial control network determines multiple sub-results of the used channels from the feature crossing result, and the sub-results of each used channel constitute the control result.

[0147] In an exemplary embodiment, the initial control network includes a mapping network and an activation network, and the feature cross result includes sub-results for multiple channels. The initial control network controls the feature cross result based on the attention processing result to obtain the control result, including: mapping the attention processing result through the mapping network to obtain the mapping result for each channel; activating the mapping result for each channel through the activation network to obtain the activation result for each channel, where the activation result characterizes the degree of channel activation; and determining the control result based on the activation result for each channel and the sub-results for multiple channels.

[0148] In this embodiment, the attention processing result is x. A hidden layer (Intermediate, hereinafter abbreviated as Inte) is used to perform feature crossing on the attention processing result x to obtain the feature crossing result. The feature crossing result includes sub-results with Ne (Ne is a positive integer) channels, using Inte... e (x) represents the sub-result of the e-th channel (e is a positive integer that takes any value from 1 to Ne).

[0149] The attention processing result x can be input into a mapping network, which then maps x to obtain the mapping results for each channel. E is then used to... e (x) represents the mapping result of the e-th channel. Next, the mapping results of each channel are input into the activation network. The activation network performs activation processing on the mapping results of each channel to obtain the predicted activation result. The predicted activation result includes the activation result of each channel, using G(E) e (x) represents the activation result of the e-th channel. The activation result of a channel characterizes its activation level; a larger activation result indicates a higher activation level. Optionally, G(E) represents the activation result of the e-th channel. e (x)) is greater than or equal to 0 and less than or equal to 1, if G(E) e If (x))=0, then the activation result of the e-th channel indicates that the e-th channel is not activated; if G(E e If (x))=1, then the activation result of the e-th channel represents the complete activation of the e-th channel. Next, the activation result of any channel is multiplied by its corresponding sub-result to obtain the multiplied result for that channel. The multiplied results of all channels are then added together to obtain the modulation result. That is, the modulation result y(x) satisfies:

[0150] In one possible implementation, the mapping network is a linear layer. The attention processing result x is weighted by the linear layer to obtain the mapping result for each channel. The mapping result for the e-th channel can be represented as w. e Alternatively, the attention processing result x can be weighted and biased through a linear layer to obtain the mapping results for each channel. The mapping result for the e-th channel can be expressed as w. e (x)+b e , where w e Characterizing the weighting function, b e Characterization bias.

[0151] Alternatively, the mapping network can be a fully connected layer. The attention processing result x is mapped through the fully connected layer to obtain the mapping results for each channel. Assuming the fully connected layer consists of two Rectified Linear Unit (ReLU) layers, the mapping result for the e-th channel can be expressed as: w e (w2(w1(x)+b1)+b2). ​​Where w e w1, w2, and w1 represent different weighting functions, and b1 and b2 represent different biases.

[0152] In one possible implementation, the activation network includes the tanh activation function. Optionally, since the tanh activation function has a value range of -1 to 1, the value range can be adjusted using (1+tanh) / 2 to make the adjusted value range between 0 and 1. Alternatively, the activation network includes the sigmoid activation function, and the sigmoid activation function has a value range between 0 and 1. The tanh activation function converges faster than the sigmoid activation function, and using the tanh activation function can better and faster activate the mapping results of each channel to the range between 0 and 1.

[0153] Suppose that the activation network includes a tanh activation function, and the range of values ​​for the activation function is adjusted using (1+tanh) / 2. Furthermore, the mapping network consists of linear layers that weight the attention processing result x. In this case, the modulation result y(x) satisfies:

[0154] It should be noted that the hidden layers in the first reconstructed network are obtained by merging the hidden layers in the first merged network corresponding to each task. For example, the hidden layers in the first reconstructed network include 3072 channels. If the network parameters of the channel corresponding to task one are on the leftmost side, and the network parameters of the channel corresponding to task one are on the rightmost side, then channels 1 to 1536 are the channels corresponding to task one (e.g., ...). Figure 4 The black area in the hidden layer shown), and channels 1537 to 3072 are the channels corresponding to Task 2 (e.g., Figure 4 (The white area in the hidden layer shown).

[0155] By determining the activation results of each channel through mapping and activation networks, and adjusting the sub-results of each channel based on these results, it is possible to activate more sub-results of channels relevant to task processing and fewer sub-results of channels irrelevant to task processing, thus avoiding conflicts between tasks. For example, if sample media data corresponding to Task 1 is input into the first task model, the mapping and activation networks can activate more sub-results of channels 1 to 1536 and fewer sub-results of channels 1537 to 3072. In this way, the features extracted by the first reconstruction network are strongly correlated with task processing, which helps improve the accuracy of prediction results and thus improves the accuracy of the multi-task model.

[0156] Furthermore, since the activation results of channels are used to characterize the activation degree of channels, adjusting the sub-results of each channel based on the activation results of each channel can ensure that the adjusted results include not only the sub-results of the channels corresponding to the target task but also the sub-results of the channels corresponding to other task processing. For example, if the sample media data corresponding to Task 1 is input into the first task model, the adjustment results determined by the mapping network and activation network will also include the sub-results of some channels from channel 1537 to 3072. Generally, different task processing may focus on the same features. Therefore, by including the sub-results of the channels corresponding to other task processing in the adjustment results, the features extracted by the first reconstruction network retain the features beneficial to the task processing, improving the representational ability of the features and thus improving the accuracy of the prediction results, thereby improving the accuracy of the multi-task model.

[0157] After determining the control result, it is input into the output layer. The control result includes sub-results after controlling multiple channels; for example, the control result may include sub-results after controlling 3072 channels. The output layer can compress these sub-results to reduce the number of channels; for example, 3072 channels can be compressed into 768 channels, resulting in multiple compressed sub-results after channel control. The output of the output layer is the output result, which includes the multiple compressed sub-results after channel control.

[0158] Step A212: Based on the output results of each first reconstruction network included in the first reconstruction network of the first layer, determine the media features output by the first reconstruction network of the first layer.

[0159] It is understandable that the first layer of the first reconstruction network includes at least one first reconstruction network, and each first reconstruction network corresponds to an output result. The output results of each first reconstruction network included in the first layer can be concatenated or summed to obtain the media features output by the first layer of the first reconstruction network. These media features are obtained by semantic-level feature extraction from the embedding vector of the sample media data and can characterize the semantics of the sample media data.

[0160] Step A22: For the non-first-layer first reconstruction network, feature extraction is performed on the media features output by the first reconstruction network above the non-first-layer first reconstruction network to obtain the media features output by the non-first-layer first reconstruction network.

[0161] Following step A21, the media features output by the first layer of the first reconstruction network can be determined. These media features are then input into the second layer of the first reconstruction network. The second layer of the first reconstruction network extracts features from these features using the same method as the first layer of the first reconstruction network for extracting features from the embedding vectors of the sample media data. This process is repeated until the media features output by the last layer of the first reconstruction network are obtained. The media features output by the second layer of the first reconstruction network are then input into the third layer of the first reconstruction network. This process is repeated again until the media features output by the last layer of the first reconstruction network are obtained. The media features output by the last layer of the first reconstruction network correspond to the sample media features mentioned above. The feature extraction method is described in step A21 and will not be repeated here.

[0162] Step A23: Based on the media features output by the first reconstruction network of the last layer, determine the prediction result corresponding to any target network model.

[0163] As mentioned above, the first task model includes the prediction network of any target network model. The media features output from the last layer of the first reconstruction network are input into the prediction network of the target network model, and the prediction result corresponding to the target network model is output.

[0164] Steps A21 to A23 describe the process of processing sample media data using the first task model from a methodological perspective. The following section will combine... Figure 6 This will be explained in detail. In the embodiments of this application, the first task model includes a character embedding layer, a model base module, and a model task head module. The model base module includes the various first reconstruction networks mentioned above, and the model task head module includes the prediction network mentioned above.

[0165] Sample media data can be input into the first task model. The sample media data includes descriptive text for describing the task processing and statement text to be decided, which is used to provide the answer to the task processing. A character embedding layer maps each character, the start character [CLS], and the separator character [SEP] in the sample media data to their corresponding character vectors. These character vectors are then concatenated to obtain a concatenated vector.

[0166] In an exemplary embodiment, the input first task model is required to have a specified number of characters, for example, 77. When the sum of the number of characters, the start character, and the separator characters in the sample media data is less than the specified number, special characters such as ##ing can be used to pad the number of characters to the specified number. The special characters can be located after each character in the sample media data and before the separator characters. A character embedding layer maps the special characters to corresponding character vectors, and the concatenation vector is determined based on these character vectors. Optionally, the character vector corresponding to the special character is an all-zero vector.

[0167] The model's basic module extracts features from the concatenated vectors to obtain the sample media features. These features include feature vectors corresponding to each character; for example, the feature vector corresponding to the start character [CLS] is [CLS]. p0 In the sample media data, each character, the special character ##ing, and the separator character [SEP] correspond to the feature vectors T1, T2, and T3 respectively. N+M .

[0168] The sample media features are input into the model task header module corresponding to a specific task, and the prediction result after processing the sample media data for that task is obtained. Taking the emotion recognition task as an example, assuming there are 8 emotion tags, the prediction result includes the probability that the sample media data belongs to each emotion tag. These 8 probabilities are cls8 to cls1.

[0169] Optionally, the concatenation vector includes the character vector corresponding to the start character [CLS]. After feature extraction of the concatenation vector by the model's base module, the feature vector corresponding to the start character is obtained, and this feature vector is used to determine the prediction result. Since the model's base module cannot obtain the prediction result information in advance, the character vector corresponding to the start character [CLS] is a vector of all zeros.

[0170] Step A3: Based on the prediction and annotation results corresponding to each target network model, train the first task model to obtain the multi-task model.

[0171] In this embodiment of the application, the target loss can be determined based on the prediction results and annotation results corresponding to each target network model. The first task model can be trained once using the target loss to obtain the trained first task model. The multi-task model can then be determined based on the trained first task model.

[0172] In an exemplary embodiment, step A3 includes: training the first task model according to the first loss and the second loss corresponding to each target network model to obtain a multi-task model.

[0173] For any target network model, the first loss is determined based on the activation results of each channel of each first reconstruction network.

[0174] As mentioned above, the hidden layers in the first reconstruction network are obtained by merging the hidden layers in the first merging network corresponding to each task processing. In other words, the hidden layers in the first reconstruction network include the network parameters of the channels corresponding to each task processing. The first reconstruction network also includes an initial modulation network, used to modulate the feature cross-output results of the hidden layers in the first reconstruction network.

[0175] Ideally, when a specific task needs to be processed, the feature cross-processing result can be manipulated through a control network. This allows the sub-results of the channel corresponding to the task being processed to be activated, while the sub-results of channels corresponding to other tasks are deactivated. By activating the task-related sub-results and suppressing the irrelevant sub-results in the feature cross-processing result, features strongly correlated with the task can be obtained.

[0176] For example, the hidden layer in the first reconstructed network includes 3072 channels. Channels 1 to 1536 correspond to Task 1, and channels 1537 to 3072 correspond to Task 2. When Task 1 is needed, the sub-results of channels 1 to 1536 are activated, while the sub-results of channels 1537 to 3072 are not activated. Similarly, when Task 2 is needed, the sub-results of channels 1 to 1536 are not activated, while the sub-results of channels 1537 to 3072 are activated.

[0177] Based on the above principles, electronic devices can determine the corresponding standard activation result for any given task. The standard activation result indicates that the channel corresponding to that task is activated, while channels corresponding to other tasks are not activated. For example, when task one is required, the standard activation result is g1 = 1…10…0 (where the first 1536 data points are 1 and the last 1536 data points are 0), indicating that channels 1 to 1536 are activated, and channels 1537 to 3072 are not activated. When task two is required, the standard activation result is g2 = 0…01…1 (where the first 1536 data points are 0 and the last 1536 data points are 1), indicating that channels 1 to 1536 are not activated, and channels 1537 to 3072 are activated.

[0178] It is understood that the first task model includes multiple first reconstruction networks, each corresponding to a predicted activation result, which includes the activation results of each channel mentioned above. Based on this, the first loss corresponding to any target network model can be determined according to the predicted activation result and the standard activation result; the method of determination is not limited here. For example, the first loss corresponding to any target network model can be determined according to any formula of Mean-Square Error (MSE), Mean Absolute Error (MAE), or Root Mean Square Error (RMSE), based on the predicted activation result and the standard activation result corresponding to each first reconstruction network.

[0179] Optionally, taking two tasks as an example, the sum of the first losses Loss3 corresponding to each target network model is determined according to the formula (1) shown below.

[0180]

[0181] Where b1 and b2 represent the weights of Task 1 and Task 2, respectively, and optionally, b1 = b2 = 0.5. k1 and k2 represent the number of sample media data corresponding to Task 1 and Task 2, respectively. i g1 represents the predicted activation results of each first reconstruction network when the i-th sample media data is processed according to Task 1 using the first task model. g1 represents the standard activation results of each first reconstruction network when the i-th sample media data is processed according to Task 1 using the first task model. j g1 represents the predicted activation results of each first reconstruction network when the j-th sample media data is processed according to the task corresponding to task 2 using the first task model. g2 represents the standard activation results of each first reconstruction network when the j-th sample media data is processed according to the task corresponding to task 2 using the first task model.

[0182] For any target network model, the second loss is determined based on the prediction and annotation results of that target network model.

[0183] In this embodiment, the second loss corresponding to any target network model can be calculated based on the prediction and annotation results of any target network model, using any formula from mean squared error, mean absolute error, root mean square error, cross-entropy loss (CE), or binary cross-entropy (BCE). Each target network model corresponds to a specific task, as shown in Examples 1 and 2 below, which illustrate the calculation process of the second loss using two task processing methods (Task 1 and Task 2) as examples.

[0184] In Example 1, Task 1 involves multi-emotion recognition of dialogue lines, corresponding to various emotion tags. As mentioned above, the prediction results for Task 1 include multiple probabilities, each representing the probability that the sample media data belongs to a different emotion tag. Furthermore, the annotation results for Task 1 include multiple labeled values, each representing whether the sample media data belongs to a different emotion tag.

[0185] As described above, for each emotion tag, the corresponding label value represents whether the sample media data belongs to that emotion tag or not. Although Task 1 corresponds to multi-emotion recognition, which is a multi-classification task, it is a binary classification task for each emotion tag. Based on this, BCE can be used to calculate the second loss for Task 1.

[0186] The BCE calculation formula is: J = -[ylog(p) + (1-y)log(1-p)], where J is the BCE calculation result, and p and y are two independent variables. Based on this BCE calculation formula, the second loss Loss1 corresponding to Task 1 can be determined according to the following formula (2).

[0187]

[0188] Where k1 represents the number of sample media data corresponding to Task 1, and n1 represents the number of emotion tags corresponding to Task 1. ij p represents the probability that the j-th sample media data belongs to the i-th sentiment label when the j-th sample media data is processed according to the task corresponding to Task 1 using the first task model. ij The characterization determines whether the j-th sample media data belongs to the i-th emotion tag when the j-th sample media data is processed according to the task corresponding to Task 1 using the first task model.

[0189] In Example 2, Task 2 involves emotion recognition from environmental description text, which corresponds to multiple emotion tags. As mentioned above, the prediction results for Task 2 include multiple probabilities, each representing the probability that the sample media data belongs to a different emotion tag. Furthermore, the annotation results for Task 1 include multiple annotation values, each representing whether the sample media data belongs to a different emotion tag.

[0190] The second loss corresponding to Task 2 can be calculated using CE. The formula for calculating CE is: Where Loss is the result of CE calculation, y i , n and n are three independent variables. Based on the calculation formula of CE, the second loss Loss2 corresponding to Task 2 can be determined according to the formula (3) shown below.

[0191]

[0192] Where k2 represents the number of sample media data corresponding to Task 2, and n2 represents the number of sentiment tags corresponding to Task 2. ij p represents the probability that the j-th sample media data belongs to the i-th sentiment tag when the j-th sample media data is processed according to the task corresponding to task two using the first task model. ij The representation indicates whether the j-th sample media data belongs to the i-th sentiment label when the j-th sample media data is processed according to the task corresponding to task two using the first task model.

[0193] After calculating the first and second losses for each target network model, the first and second losses can be summed, averaged, or weighted to obtain the target loss. For example, the target loss Loss can be calculated as Loss = Loss1 + Loss2 + aLoss31 + aLoss32, where Loss1 is the second loss for task one, Loss2 is the second loss for task two, Loss31 is the first loss for task one, Loss32 is the first loss for task two, and 'a' is a hyperparameter, optionally a = 0.2. Then, the first task model is trained once using the target loss to obtain the trained first task model, and the multi-task model is determined based on the trained first task model.

[0194] Understandably, each task processing corresponds to multiple sample media data. Assuming a task processing corresponds to N sample media data, each set of media data (bs) can be input into the model as a batch. In this case, the task processing corresponds to N / bs batches of data. After inputting all N / bs batches of data into the model, we can obtain N output results, each including the prediction result and prediction activation result mentioned above. During each training process, the target loss needs to be determined using the N results corresponding to each task processing, and the model is trained once using the target loss.

[0195] In implementation method B, step 204 includes steps B1 to B2 (not shown in the figure).

[0196] Step B1 involves training each initial control network in the first task model to obtain the individual first control networks. By training only the initial control networks, fine-tuning of the model is achieved, avoiding the introduction of new learning convergence problems caused by adjusting all network parameters of the model.

[0197] In this embodiment, the model can undergo a first stage of training. During this first stage, the network parameters of the initial control network are set to be trainable, while the network parameters of other parts, including the first merging network, are set to be non-trainable. Following the implementation principle described in implementation method A, a first loss and / or a second loss corresponding to each target network model are determined based on the first task model, and these losses are used to calculate the first-stage loss. Each initial control network is then trained once using the first-stage loss to obtain each trained initial control network.

[0198] If the initial control networks after training meet the training termination conditions of the first stage, then each initial control network after training is used as a first control network. If the initial control networks after training do not meet the training termination conditions of the first stage, then each initial control network after training is used as an initial control network for the next training iteration. The initial control networks are then trained again according to the training method of the first stage to obtain each initial control network after training. Based on each initial control network after training, a first control network is determined.

[0199] Understandably, each task processing involves multiple sample media data sets. Assuming a task processing involves N sample media data sets, each set of media data (bs) can be input into the model as a batch. Therefore, this task processing corresponds to N / bs batches. All N / bs batches are input into the model, and the model is trained based on its output. Each task processing requires batch input into the model.

[0200] At this point, the first task model is updated to the second task model, which includes multiple second reconstructed networks forming a topological connection structure. Each second reconstructed network includes a first merging network and a first control network. It is understood that the second task model and the first task model have similar structures, differing only in their network parameters.

[0201] This application does not limit the initial control network after training to meeting the training termination conditions of the first stage. For example, the initial control network after training meeting the training termination conditions of the first stage includes: the number of training iterations of the initial control network after training reaches a set number, or the error of the initial control network after training is within a set error range, etc.

[0202] By training each initial control network to obtain each first control network, the control capability of the first control network can be improved. This allows the first control network to accurately activate features related to task processing and suppress features unrelated to task processing, avoiding feature conflicts between different task processing tasks. This enables the model to have the ability to process different tasks and ensures the accuracy of task processing results.

[0203] Step B2 involves alternating training of each first merging network and each first control network to obtain a multi-task model.

[0204] In this embodiment, the model can undergo a second stage of training. This second stage is an alternating training phase, comprising two sub-stages. In the first sub-stage, the network parameters of the first control network are set to be trainable, while the network parameters of other parts, including the first merging network, are set to be non-trainable. In the second sub-stage, the network parameters of the first merging network are set to be trainable, while the network parameters of other parts, including the first control network, are set to be non-trainable.

[0205] For the first sub-stage, following the implementation principle described in implementation method A, the first loss and / or second loss corresponding to each target network model are determined based on the second task model, and these losses are used to calculate the loss of the first sub-stage. Each first regulation network is then trained once using the loss from the first sub-stage, resulting in each trained first regulation network.

[0206] At this point, the second task model is updated to the third task model, which includes multiple third-reconstruction networks forming a topological connection structure. Each third-reconstruction network includes a first merging network and a trained first regulation network. It is understandable that the third task model and the first task model have similar structures, differing only in their network parameters.

[0207] For the second sub-stage, following the implementation principle described in implementation method A, the first loss and / or second loss corresponding to each target network model are determined based on the third task model, and these losses are used to calculate the loss of the second sub-stage. The first merging networks are then trained once using the losses from the second sub-stage to obtain the trained first merging networks.

[0208] At this point, the third task model is updated to the fourth task model, which includes multiple fourth reconstruction networks forming a topological connection structure. Each fourth reconstruction network includes the trained first merging network and the trained first regulation network. It is understandable that the fourth task model and the first task model have similar structures, differing only in their network parameters.

[0209] If each trained first merging network and each trained first regulating network satisfies the training termination condition of the second stage (i.e., the fourth task model satisfies the training termination condition of the second stage), then each trained first regulating network is used as a target regulating network, and each trained first merging network is used as a target merging network. In other words, the fourth task model is treated as a multi-task model, and each fourth reconstruction network is used as a target reconstruction network. The target reconstruction network includes the target regulating network and the target merging network.

[0210] If the fourth task model does not meet the training termination condition of the second stage, then the fourth task model is used as the second task model for the next training. Following the training method of the second stage, the first control network and the first merging network in the second task model are trained alternately to obtain the fourth task model, and the multi-task model is determined based on the fourth task model.

[0211] This application does not limit the fourth task model to satisfy the training termination conditions of the second stage. For example, the training termination conditions of the fourth task model in the second stage include: the number of training sessions in the second stage reaches a set number, or the error of the fourth task model is within a set error range, etc.

[0212] In this embodiment, on the one hand, a first merging network is trained using the first loss and / or second loss corresponding to each task processing. This allows the trained first merging network to extract features of interest to each task processing, improving the representational ability of the features and thus enhancing the accuracy of the model. On the other hand, a first regulation network is trained using the first loss and / or second loss corresponding to each task processing. This allows the trained first regulation network to accurately activate features related to a specific task processing and suppress features unrelated to that task processing, avoiding feature conflicts between different task processing. This enables the model to handle different tasks while ensuring the accuracy of the task processing results.

[0213] By alternately training the first merging network and the first regulation network using the first loss and / or second loss corresponding to each task, information fusion between different task processes is achieved. This allows any task process to learn useful information from other task processes, improving the representational ability of the features extracted by the model and their correlation with the task processes, thereby improving the accuracy of the model.

[0214] Optionally, the first task model also includes prediction networks for each target network model. In at least one of the first stage, the first sub-stage of the second stage, or the second sub-stage, the prediction networks for each target network model may or may not be trained; this is not limited here.

[0215] In practical applications, in addition to the first and second stages of training mentioned above, there can be more training stages. For example, after completing the second stage of training and obtaining the fifth task model, a third stage of training can be performed on the fifth task model. During the third stage of training, all networks of the fifth task model (including the control network, merging network, prediction network, etc.) are trained, and a multi-task model is obtained based on the trained model. The third stage of training can use a relatively low learning rate, such as 0.0005, and the stochastic gradient descent method is used for training based on this learning rate.

[0216] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant regions. For example, the target network model, sample media data, and annotation results involved in this application were all obtained with full authorization.

[0217] In the above method, each target network model includes multiple feature processing networks, and the topological connection structure formed by these feature processing networks is identical across all target network models. By merging the feature processing networks located at the same position in the topological connection structure, first merged networks are obtained, which transfers the network parameters of the feature processing networks to the first merged networks, thus accelerating the training efficiency of the first merged networks. A first task model is constructed using these first merged networks, and a multi-task model is obtained by training the first task model. This enables rapid training of the multi-task model for task processing, improving task processing efficiency.

[0218] like Figure 7 As shown, Figure 7This application provides a method for processing media data, which can be applied to the above-described implementation environment. The method in this application is executed by an electronic device, including a terminal device 101 or a server 102, and includes the following steps.

[0219] Step 701: Obtain target media data.

[0220] The content and acquisition methods of the target media data can be found in the section on the content and acquisition methods of the sample media data above. The implementation principles of the two are similar and will not be repeated here.

[0221] Step 702: Perform at least one task on the target media data using a multi-task model to obtain the results of each task.

[0222] Among them, the multi-task model is based on and Figure 2 The relevant multi-task model was trained using a specific training method. The method of processing target media data using a multi-task model is similar to the method of processing sample media data using the first-task model described above; their implementation principles are similar and will not be repeated here.

[0223] As can be seen from the training method of the multi-task model described above, the multi-task model in this embodiment includes multiple target reconstruction networks forming a topological connection structure and prediction networks corresponding to each task processing. Optionally, after receiving target media data and at least one task processing input by the user, the electronic device invokes the multi-task model. First, feature extraction is performed on the target media data through multiple target reconstruction networks to obtain target media features, which are used to characterize the semantics of the target media data. Then, for any task processing input by the user, the task processing result is determined by the prediction network corresponding to that task processing based on the target media features.

[0224] like Figure 8 As shown, the user inputs Task 1 and the test data XXX. The service system can call the model base module and the model task header corresponding to Task 1 from the multi-task model. The model base module includes multiple target reconstruction networks, and the model task header corresponding to Task 1 includes the prediction network for Task 1. Based on the model base module and the model task header corresponding to Task 1 called by the service system, the label XXXXX of the test data is determined. This label is the task processing result mentioned above.

[0225] The embodiments of this application can, for certain task processing, only call the prediction network related to the task processing to participate in the calculation, and do not call the prediction network unrelated to the task processing, thereby reducing the amount of computation and the computation time, and improving the speed of multi-task model in task processing.

[0226] In one possible implementation, the multi-task model includes a multi-layer target reconstruction network; step 702 includes: for the first-layer target reconstruction network, feature extraction is performed on the target media data through the first-layer target reconstruction network to obtain the media features output by the first-layer target reconstruction network; for non-first-layer target reconstruction networks, feature extraction is performed on the media features output by the target reconstruction network above the non-first-layer target reconstruction network through the non-first-layer target reconstruction network to obtain the media features output by the non-first-layer target reconstruction network; based on the media features output by the last-layer target reconstruction network, the processing results of each task are determined.

[0227] The implementation principle of the above steps is similar to that described in steps A21 to A23, as can be seen in the descriptions of steps A21 to A23, and will not be repeated here.

[0228] It should be noted that all information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, stored data, displayed data, etc.), and signals involved in this application have been authorized by the user or fully authorized by all parties, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant regions. For example, the multi-task model and target media data involved in this application were obtained with full authorization.

[0229] The above-mentioned method has high training efficiency for multi-task models, which enables efficient task processing of target media data through multi-task models.

[0230] The above describes the training method for the multi-task model or the media data processing method provided in the embodiments of this application from the perspective of method steps. The following is a systematic and comprehensive explanation. The method of the embodiments of this application can be applied to scenarios involving at least two task processing tasks, where each task processing involves processing media data such as text, video, audio, and images, such as classification, emotion recognition, and translation. The following uses two emotion recognition tasks as examples, combined with... Figure 9 The method of this application embodiment will be described below. Among them, emotion recognition task 1 is to perform emotion recognition on dialogue text. Emotion recognition task 2 is to perform emotion recognition on environmental description text. For example... Figure 9 As shown, the method in this application embodiment includes the following steps.

[0231] Step 901: Obtain emotion recognition model 1, dialogue text and emotion annotation result 1, and obtain emotion recognition model 2, environment description text and emotion annotation result 2.

[0232] The implementation method of step 901 can be found in the description of steps 201 and A1. The implementation principles of the two are similar, and will not be repeated here.

[0233] Step 902: Merge the Transformer structure in emotion recognition model 1 and the Transformer structure in emotion recognition model 2 to obtain a first merged network. Based on the first merged network and the initial control network, construct a first reconstruction network. The first task model includes the first reconstruction network.

[0234] The implementation method of step 902 can be found in the description of steps 202 and 203. The implementation principles of the two are similar, and will not be repeated here.

[0235] Step 903: Input the dialogue text into the first task model to obtain the predicted activation result 1 determined by the Transformer structure in the first task model and the emotion prediction result 1-1 output by the first task model.

[0236] The implementation method of step 903 can be found in the description of step A2. The implementation principles of the two are similar, and will not be repeated here.

[0237] Step 904: Determine the first loss 1 based on the predicted activation result 1 and the standard activation result 1, and determine the second loss 1-1 based on the emotion prediction result 1-1 and the emotion labeling result 1.

[0238] The method for determining the first loss 1 can be found in step A3 regarding the first loss, and the method for determining the second loss 1-1 can be found in step A3 regarding the second loss. The implementation principles of the two are similar, and will not be repeated here.

[0239] Step 905: Input the environmental description text into the first task model to obtain the predicted activation result 2 determined by the Transformer structure in the first task model and the emotion prediction results 1-2 output by the first task model.

[0240] The implementation method of step 905 can be found in the description of step A2. The implementation principles of the two are similar, and will not be repeated here.

[0241] Step 906: Determine the first loss 2 based on the predicted activation result 2 and the standard activation result 2, and determine the second loss 1-2 based on the emotion prediction result 1-2 and the emotion labeling result 2.

[0242] The method for determining the first loss 2 can be found in the content related to the first loss in step A3, and the method for determining the second loss 1-2 can be found in the content related to the second loss in step A3. The implementation principles of the two are similar, and will not be repeated here.

[0243] Step 907: Train the initial regulation network in the first task model using the first loss 1, the first loss 2, the second loss 1-1, and the second loss 1-2 to obtain the first regulation network. The second task model includes the first regulation network and the first merging network.

[0244] The implementation method of step 907 can be found in the description of step B1. The implementation principles of the two are similar, and will not be repeated here.

[0245] Steps 903 to 907 above correspond to the first stage of training. The following section combines... Figure 10 Let me elaborate on the training process in the first phase.

[0246] In this embodiment of the application, the first task model includes first reconstruction networks Eb1 to Eb12, and one first reconstruction network is... Figure 10 The expert Transformer structure is shown. The expert Transformer structure includes a first merging network and an initial control network. The first merging network includes an attention layer, a hidden layer, and an output layer. The initial control network includes an expert module and a gating module.

[0247] Both emotion recognition model 1 and emotion recognition model 2 include a Transformer structure, which consists of an attention layer, a hidden layer, and an output layer. The attention layer in the expert Transformer structure is the same as the attention layer in the Transformer structure of emotion recognition model 1 (e.g., ...). Figure 10 The black portion of the attention layer shown) and the attention layer included in the Transformer structure of emotion recognition model 2 (such as...) Figure 10 The hidden layers (shown as white portions in the attention layer) are merged to obtain the result. Similarly, the hidden layers included in the expert Transformer structure are derived from the hidden layers included in the Transformer structure of emotion recognition model 1 (such as...). Figure 10 The black parts in the hidden layer shown) and the hidden layers included in the Transformer structure of emotion recognition model 2 (such as the black parts in the hidden layer shown) and emotion recognition model 2. Figure 10 The white portion of the hidden layer shown is merged to obtain the output layer of the expert Transformer structure (such as the output layer of the Transformer structure in emotion recognition model 1). Figure 10 The black portion of the output layer shown) and the output layer (such as) included in the Transformer structure of emotion recognition model 2 in the emotion recognition model 2 Figure 10 The white portion of the output layer shown is merged to obtain the result.

[0248] Optionally, there are N dialogue texts and N environment description texts. For each N dialogue text, every bs of dialogue text is input into the model as a batch of data. Similarly, for each N environment description text, every bs of environment description text is input into the model as a batch of data. The subsequent processing of each text is similar. The following describes the subsequent processing using one dialogue text and one environment description text as an example.

[0249] On one hand, the dialogue text is input into the first task model, and features are extracted from the dialogue text through the first reconstruction network Eb1 to Eb12 to obtain the first dialogue text features, which are used to describe the semantics of the dialogue text. During the feature extraction process, the activation results of each channel in the output of the hidden layer are determined by the expert module. The predicted activation result 1 includes the activation results of each channel, and the predicted activation result 1 is used together with the standard activation result 1 to determine the first loss 1.

[0250] The first task model also includes a Sigmoid network, which is the prediction network in the emotion recognition model 1. The Sigmoid network maps the features of the first dialogue text to the emotion prediction result 1-1, and the emotion prediction result 1-1 is used together with the emotion annotation result 1 of the dialogue text to determine the second loss 1-1.

[0251] On the other hand, the environmental description text is input into the first task model, and features are extracted from the environmental description text through the first reconstruction network Eb1 to Eb12 to obtain the first environmental description text features. The first environmental description text features are used to describe the semantics of the environmental description text. During the feature extraction process, the activation results of each channel in the output of the hidden layer are determined by the expert module. The predicted activation result 2 includes the activation results of each channel, and the predicted activation result 2 is used together with the standard activation result 2 to determine the first loss 2.

[0252] The first task model also includes a Softmax network, which is the prediction network in the emotion recognition model 2. The Softmax network maps the features of the first environment description text to emotion prediction results 1-2. The emotion prediction results 1-2 are used together with the emotion annotation results 2 of the environment description text to determine the second loss 1-2.

[0253] Next, the expert module and the gating module are trained using the first loss 1, the first loss 2, the second loss 1-1, and the second loss 1-2 to complete one training cycle.

[0254] Optionally, in each training iteration of the first phase, in addition to training the expert module and the gating module, a Sigmoid network and a Softmax network can also be trained. Learning parameters can be set, such as a learning rate of 0.005, using the Adam optimizer for training, setting the learning rate to 0.1 times its original value every 5 training iterations, and setting the number of training iterations to 10. After multiple training iterations based on the learning parameters, the first task model is updated to the second task model.

[0255] Step 908: Input the dialogue text into the second task model to obtain the emotion prediction result 2-1. Based on the emotion prediction result 2-1 and the emotion labeling result 1, determine the second loss 2-1.

[0256] The implementation method of step 908 can be found in the content of steps A2 and A3. The implementation principles of the two are similar, and will not be repeated here.

[0257] Step 909: Input the environmental description text into the second task model to obtain the emotion prediction result 2-2. Based on the emotion prediction result 2-2 and the emotion annotation result 2, determine the second loss 2-2.

[0258] The implementation method of step 909 can be found in the content of steps A2 and A3. The implementation principles of the two are similar, and will not be repeated here.

[0259] Step 910: Train the first regulation network in the second task model using the second loss 2-1 and the second loss 2-2. The third task model includes the trained first regulation network and the first merging network.

[0260] The implementation method of step 910 can be found in the content related to the first sub-stage in step B2. The implementation principles of the two are similar, so they will not be repeated here.

[0261] Steps 908 to 910 above correspond to the first sub-stage in the second stage of training. The following section combines... Figure 11 This section will elaborate on the training process of the first sub-stage. Among them, Figure 11 The second task model shown is based on Figure 10 The training process shown is to train the first task model, and the second task model includes a second reconstruction network that is trained from the first reconstruction network.

[0262] On one hand, the dialogue text is input into the second task model, and features are extracted from the dialogue text through the second reconstruction network Eb1 to Eb12 to obtain the second dialogue text features, which are used to describe the semantics of the dialogue text. The second dialogue text features are mapped to the sentiment prediction result 2-1 through the Sigmoid network. The sentiment prediction result 2-1 is used together with the sentiment annotation result 1 of the dialogue text to determine the second loss 2-1.

[0263] On the other hand, the environmental description text is input into the second task model, and features are extracted from the environmental description text through the second reconstruction network Eb1 to Eb12 to obtain the second environmental description text features. The second environmental description text features are used to describe the semantics of the environmental description text. The second environmental description text features are mapped to the sentiment prediction result 2-2 through the Softmax network. The sentiment prediction result 2-2 is used together with the sentiment annotation result 2 of the environmental description text to determine the second loss 2-2.

[0264] Next, the expert module and the gating module are trained using the second loss 2-1 and the second loss 2-2. After one training cycle, the second task model is updated to the third task model.

[0265] Step 911: Input the dialogue text into the third task model to obtain the emotion prediction result 3-1. Based on the emotion prediction result 3-1 and the emotion labeling result 1, determine the second loss 3-1.

[0266] The implementation method of step 911 can be found in the content of steps A2 and A3. The implementation principles of the two are similar, and will not be repeated here.

[0267] Step 912: Input the environmental description text into the second task model to obtain the emotion prediction result 3-2. Based on the emotion prediction result 3-2 and the emotion annotation result 2, determine the second loss 3-2.

[0268] The implementation method of step 912 can be found in the content of steps A2 and A3. The implementation principles of the two are similar, and will not be repeated here.

[0269] Step 913: Train the first merging network using the second loss 3-1 and the second loss 3-2. The fourth task model includes the trained first regulation network and the trained first merging network.

[0270] The implementation method of step 913 can be found in the content related to the second sub-stage in step B2. The implementation principles of the two are similar, and will not be repeated here.

[0271] Steps 911 to 913 above correspond to the second sub-stage in the second phase of training. The following section combines... Figure 11 Let's elaborate on the training process of the second sub-stage. Among them, Figure 11 The third task model shown is based on Figure 11 The training process of the first sub-stage shown is to train the second task model, and the third task model, including the third reconstruction network, is to train the second reconstruction network.

[0272] On one hand, the dialogue text is input into the third task model, and features are extracted from the dialogue text through the third reconstruction network Eb1 to Eb12 to obtain the third dialogue text features, which are used to describe the semantics of the dialogue text. The third dialogue text features are mapped to the sentiment prediction result 3-1 through the Sigmoid network. The sentiment prediction result 3-1 is used together with the sentiment annotation result 1 of the dialogue text to determine the second loss 3-1.

[0273] On the other hand, the environmental description text is input into the third task model, and features are extracted from the environmental description text through the third reconstruction network Eb1 to Eb12 to obtain the third environmental description text features. The third environmental description text features are used to describe the semantics of the environmental description text. The third environmental description text features are mapped to the sentiment prediction result 3-2 through the Softmax network. The sentiment prediction result 3-2 is used together with the sentiment annotation result 2 of the environmental description text to determine the second loss 3-2.

[0274] Next, the hidden layers are trained using the second loss 3-1 and the second loss 3-2. After one training cycle, the third task model is updated to the fourth task model.

[0275] The fourth task model can be used as the second task model, and steps 908 to 913 can be executed cyclically. That is, the second phase of training includes alternating execution of the first and second sub-phases.

[0276] Optionally, during each training iteration of the first and second sub-stages, both the Sigmoid and Softmax networks can be trained. Learning parameters can be set, such as a learning rate of 0.005, using SGD (Stochastic Gradient Descent) for training, setting the learning rate to 0.1 times the original value every 5 training iterations, and setting the number of training iterations to 30. After multiple alternating training iterations based on these learning parameters, a multi-task model is obtained.

[0277] After training the multi-task model, you only need to input the test data and call the multi-task model through a service system. The multi-task model can then process the test data into multiple tasks and obtain the results of each task.

[0278] If you need to output the processing results of a specific task, such as Figure 8As shown, a task can be selected, and the service system can call the model base modules (i.e., Eb1 to Eb12 in the multi-task model) and the model task header modules (i.e., the Sigmoid network or Softmax network in the multi-task model) in the multi-task model. This allows the multi-task model to process the test data for a specific task and obtain the task processing result. This avoids calling irrelevant model task header modules, reduces inference costs, and improves model computation efficiency.

[0279] In this embodiment, on the one hand, a multi-task model can be used to perform emotion recognition on the dialogue text to obtain emotion recognition results. Subsequently, various processing can be performed based on these emotion recognition results. For example, the emotion recognition results of the dialogue text can be displayed on an electronic device, or a prompt message matching the emotion recognition results of the dialogue text can be generated. For instance, if the emotion recognition results include fatigue, a prompt message can be generated to remind the user to rest or play relaxing music.

[0280] On the other hand, a multi-task model can be used to perform emotion recognition on the environmental description text, yielding emotion recognition results. These results can then be used for various processing steps. For example, the emotion recognition result of the environmental description text is closely related to the scene setting. Based on this, environmental scene suggestions matching the emotion recognition result of the environmental description text can be generated. If the emotion recognition result of the environmental description text indicates a cool or warm color scheme, suggestions could include adding fluffy or stiff accessories to neutralize the color scheme, or suggesting that the lighting in the scene matches the color scheme, or recommending music that conveys the color scheme (such as light, heavy, or solemn music) to better express the environment. Conversely, if the emotion recognition result of the environmental description text indicates enthusiasm, suggestions could include using red silk, warm lighting, and bright, upbeat music to enhance the scene's atmosphere.

[0281] Figure 12 The diagram shown is a structural schematic of a training device for a multi-task model provided in an embodiment of this application. Figure 12 As shown, the device includes:

[0282] The acquisition module 1201 is used to acquire at least two target network models. Each target network model includes multiple feature processing networks. The topological connection structure formed by the multiple feature processing networks included in each target network model is the same. Each target network model is used to perform a task processing on the target media data.

[0283] The merging module 1202 is used to merge the feature processing networks located at the same position in the topological connection structure of at least two target network models, and merge the feature processing networks at one position to obtain a first merged network.

[0284] Module 1203 is used to construct the first task model based on each first merging network;

[0285] Training module 1204 is used to train the first task model to obtain a multi-task model. The multi-task model is used to process the target media data using at least one of the task processing methods corresponding to each target network model.

[0286] In one possible implementation, any feature processing network includes a multi-layer processing sub-network, any layer processing sub-network includes network parameters for multiple channels, and a first merging network includes a multi-layer merging sub-network.

[0287] The merging module 1202 is used to select network parameters of a specified channel from the network parameters of each channel of any processing sub-network included in any feature processing network; and to construct a merged sub-network based on the network parameters of the specified channels of multiple processing sub-networks at a given location.

[0288] In one possible implementation, the merging module 1202 is used to normalize the network parameters of each channel included in any layer of the processing sub-network to obtain the normalized network parameters of each channel; based on the normalized network parameters of each channel, determine the importance index of each channel; and determine a specified channel from each channel whose importance index is not less than the index threshold, and obtain the network parameters of the specified channel.

[0289] In one possible implementation, the construction module 1203 is used to construct a first reconstruction network based on any first merging network and an initial control network for any first merging network, wherein the initial control network is used to control the output of the first reconstruction network; and to construct a first task model based on each first reconstruction network.

[0290] In one possible implementation, the training module 1204 is used to acquire sample media data and annotation results for each target network model. The sample media data corresponding to any target network model is used to train any target network model, and the annotation results of any target network model are used to annotate the processing results obtained by performing task processing on the sample media data. For the sample media data of any target network model, the first task model is used to perform corresponding task processing on the sample media data of any target network model to obtain the prediction result corresponding to any target network model. Based on the prediction results and annotation results corresponding to each target network model, the first task model is trained to obtain a multi-task model.

[0291] In one possible implementation, the first reconstruction networks are arranged in multiple layers;

[0292] Training module 1204 is used to extract features from sample media data of any target network model through the first reconstruction network of the first layer, and obtain the media features output by the first reconstruction network of the first layer; for non-first reconstruction networks, it extracts features from the media features output by the first reconstruction network of the previous layer of the non-first reconstruction network, and obtains the media features output by the non-first reconstruction network; based on the media features output by the last first reconstruction network, it determines the prediction result corresponding to any target network model.

[0293] In one possible implementation, the first layer of the first reconstruction network includes multiple first reconstruction networks, and the first merging network in the first reconstruction network includes an attention layer, a hidden layer, and an output layer;

[0294] The training module 1204 is used to perform attention processing on the sample media data of any target network model through an attention layer for any first reconstruction network included in the first layer first reconstruction network, to obtain the attention processing result; to perform feature crossing on the attention processing result through a hidden layer, to obtain the feature crossing result; to adjust the feature crossing result based on the attention processing result through an initial adjustment network, to obtain the adjustment result; to determine the output result based on the adjustment result through an output layer; and to determine the media features output by the first layer first reconstruction network based on the output results of each first reconstruction network included in the first layer first reconstruction network.

[0295] In one possible implementation, the initial control network includes a mapping network and an activation network, and the feature cross-results include sub-results from multiple channels;

[0296] Training module 1204 is used to map the attention processing results through a mapping network to obtain the mapping results of each channel; to activate the mapping results of each channel through an activation network to obtain the activation results of each channel, and the activation results of the channels are used to characterize the activation degree of the channels; and to determine the regulation result based on the activation results of each channel and the sub-results of multiple channels.

[0297] In one possible implementation, the training module 1204 is used to train the first task model according to the first loss and the second loss corresponding to each target network model to obtain a multi-task model.

[0298] The first loss corresponding to any target network model is determined based on the activation results of each channel corresponding to each first reconstruction network.

[0299] The second loss corresponding to any target network model is determined based on the prediction and annotation results of that target network model.

[0300] In one possible implementation, the training module 1204 is used to train each initial control network in the first task model to obtain each first control network; and to alternately train each first merging network and each first control network to obtain a multi-task model.

[0301] In the aforementioned apparatus, each target network model includes multiple feature processing networks, and the topological connection structure formed by these feature processing networks is identical across all target network models. By merging the feature processing networks located at the same position in the topological connection structure, first merged networks are obtained, enabling the transfer of network parameters from the feature processing networks to the first merged networks, which facilitates faster training of the first merged networks. A first task model is constructed using these first merged networks, and a multi-task model is obtained by training the first task model. This allows for rapid training of the multi-task model, enabling task processing to be performed using the multi-task model and improving task processing efficiency.

[0302] It should be understood that the above Figure 12 The provided device, in implementing its functions, is only illustrated by the division of the above-described functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.

[0303] Figure 13 The diagram shown is a structural schematic of a media data processing device provided in an embodiment of this application. Figure 13 As shown, the device includes:

[0304] Module 1301 is used to acquire target media data;

[0305] Task processing module 1302 is used to perform at least one task processing on the target media data through a multi-task model, and obtain the results of each task processing. The multi-task model is based on... Figure 2 The relevant multi-task model was trained using training methods.

[0306] In one possible implementation, the multi-task model includes a multi-layer target reconstruction network;

[0307] The task processing module 1302 is used to extract features from the target media data through the first-layer target reconstruction network to obtain the media features output by the first-layer target reconstruction network; for non-first-layer target reconstruction networks, it extracts features from the media features output by the target reconstruction network above the non-first-layer target reconstruction network to obtain the media features output by the non-first-layer target reconstruction network; and determines the processing results of each task based on the media features output by the last-layer target reconstruction network.

[0308] The multi-task model in the aforementioned device has high training efficiency, enabling efficient task processing of target media data through the multi-task model.

[0309] It should be understood that the above Figure 13 The provided device, in implementing its functions, is only illustrated by the division of the above-described functional modules. In practical applications, the functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. Furthermore, the device and method embodiments provided in the above embodiments belong to the same concept, and their specific implementation processes are detailed in the method embodiments, and will not be repeated here.

[0310] Figure 14 A structural block diagram of a terminal device 1400 provided in an exemplary embodiment of this application is shown. The terminal device 1400 includes a processor 1401 and a memory 1402.

[0311] Processor 1401 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1401 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1401 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1401 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content required to be displayed on the screen. In some embodiments, processor 1401 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0312] The memory 1402 may include one or more computer-readable storage media, which may be non-transitory. The memory 1402 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1402 are used to store at least one computer program, which is executed by the processor 1401 to implement the training method of the multi-task model or the media data processing method provided in the method embodiments of this application.

[0313] In some embodiments, the terminal device 1400 may also optionally include a peripheral device interface 1403 and at least one peripheral device. The processor 1401, memory 1402, and peripheral device interface 1403 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1403 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1404, a display screen 1405, a camera assembly 1406, an audio circuit 1407, and a power supply 1408.

[0314] Peripheral device interface 1403 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1401 and memory 1402. In some embodiments, processor 1401, memory 1402 and peripheral device interface 1403 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1401, memory 1402 and peripheral device interface 1403 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0315] The radio frequency (RF) circuit 1404 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1404 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1404 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. Optionally, the RF circuit 1404 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1404 can communicate with other terminals through at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1404 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0316] Display screen 1405 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1405 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1401 for processing. In this case, display screen 1405 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1405, disposed on the front panel of terminal device 1400; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal device 1400 or in a folded design; in still other embodiments, display screen 1405 may be a flexible display screen, disposed on a curved or folded surface of terminal device 1400. Furthermore, display screen 1405 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1405 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0317] The camera assembly 1406 is used to acquire images or videos. Optionally, the camera assembly 1406 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1406 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0318] The audio circuit 1407 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1401 for processing, or input to the radio frequency circuit 1404 to achieve voice communication. For stereo sound acquisition or noise reduction purposes, there may be multiple microphones, each located at a different part of the terminal device 1400. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1401 or the radio frequency circuit 1404 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1407 may also include a headphone jack.

[0319] Power supply 1408 is used to power the various components in terminal device 1400. Power supply 1408 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1408 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0320] In some embodiments, the terminal device 1400 further includes one or more sensors 1409. The one or more sensors 1409 include, but are not limited to: an accelerometer 1411, a gyroscope 1412, a pressure sensor 1413, an optical sensor 1414, and a proximity sensor 1415.

[0321] Accelerometer 1411 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established by terminal device 1400. For example, accelerometer 1411 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1401 can control display screen 1405 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1411. Accelerometer 1411 can also be used for games or for acquiring user motion data.

[0322] The gyroscope sensor 1412 can detect the orientation and rotation angle of the terminal device 1400. The gyroscope sensor 1412 can work in conjunction with the accelerometer sensor 1411 to collect 3D motion data from the user on the terminal device 1400. Based on the data collected by the gyroscope sensor 1412, the processor 1401 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0323] The pressure sensor 1413 can be disposed on the side bezel of the terminal device 1400 and / or on the lower layer of the display screen 1405. When the pressure sensor 1413 is disposed on the side bezel of the terminal device 1400, it can detect the user's grip signal on the terminal device 1400, and the processor 1401 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1413. When the pressure sensor 1413 is disposed on the lower layer of the display screen 1405, the processor 1401 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1405. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0324] Optical sensor 1414 is used to collect ambient light intensity. In one embodiment, processor 1401 can control the display brightness of display screen 1405 based on the ambient light intensity collected by optical sensor 1414. Specifically, when the ambient light intensity is high, the display brightness of display screen 1405 is increased; when the ambient light intensity is low, the display brightness of display screen 1405 is decreased. In another embodiment, processor 1401 can also dynamically adjust the shooting parameters of camera assembly 1406 based on the ambient light intensity collected by optical sensor 1414.

[0325] The proximity sensor 1415, also known as a distance sensor, is typically located on the front panel of the terminal device 1400. The proximity sensor 1415 is used to detect the distance between the user and the front of the terminal device 1400. In one embodiment, when the proximity sensor 1415 detects that the distance between the user and the front of the terminal device 1400 is gradually decreasing, the processor 1401 controls the display screen 1405 to switch from a screen-on state to a screen-off state; when the proximity sensor 1415 detects that the distance between the user and the front of the terminal device 1400 is gradually increasing, the processor 1401 controls the display screen 1405 to switch from a screen-off state to a screen-on state.

[0326] Those skilled in the art will understand that Figure 14 The structure shown does not constitute a limitation on the terminal device 1400, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0327] Figure 15This is a schematic diagram of the server structure provided in the embodiments of this application. The server 1500 can vary considerably due to different configurations or performance. It may include one or more processors 1501 and one or more memories 1502. The one or more memories 1502 store at least one computer program, which is loaded and executed by the one or more processors 1501 to implement the training method of the multi-task model or the media data processing method provided in the above-described method embodiments. For example, the processor 1501 is a CPU. Of course, the server 1500 may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server 1500 may also include other components for implementing device functions, which will not be elaborated here.

[0328] In an exemplary embodiment, a computer-readable storage medium is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement the training method of any of the above-described multi-task models or the media data processing method.

[0329] Optionally, the aforementioned computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0330] In an exemplary embodiment, a computer program is also provided, which is at least one such computer program, loaded and executed by a processor, to enable an electronic device to implement the training method of any of the above-described multi-task models or the media data processing method.

[0331] In an exemplary embodiment, a computer program product is also provided, which stores at least one computer program that is loaded and executed by a processor to enable an electronic device to implement the training method of any of the above-described multi-task models or the media data processing method.

[0332] It should be understood that "multiple" as used in this article refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0333] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0334] The above description is merely an exemplary embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the principles of this application should be included within the protection scope of this application.

Claims

1. A training method for a multi-task model, characterized in that, The method includes: Obtain at least two target network models, each target network model including multiple feature processing networks, each feature processing network including multi-layer processing sub-networks, each layer processing sub-network including network parameters of multiple channels, the topological connection structure formed by the multiple feature processing networks included in each target network model is the same, each target network model is used to perform a task processing on target media data, the target media data including at least one of text, image, audio, and video; The feature processing networks located at the same position in the topological connection structure of the at least two target network models are merged, and the feature processing networks at one position are merged to obtain a first merged network; wherein, the first merged network includes multi-layer merged sub-networks, and each merged sub-network includes network parameters of a specified channel in any layer of the feature processing network located at the same position whose information entropy is not less than an index threshold. Based on each of the first merged networks, construct the first task model; Obtain sample media data and annotation results for each target network model. The sample media data includes at least one of text, image, audio, and video. The sample media data of any target network model is used to train the target network model, and the annotation results of the target network model are used to annotate the processing results obtained by processing the sample media data. For any of the target network models' sample media data, the first task model is used to perform corresponding task processing on the sample media data of any of the target network models to obtain the prediction result corresponding to any of the target network models. The first loss corresponding to any target network model is determined based on the predicted activation result and the standard activation result of each specified channel in the first task model. The predicted activation result of the specified channel represents the activation degree of the specified channel, and the standard activation result of the specified channel represents whether the specified channel corresponding to any target network model is activated or the specified channel corresponding to other target network models is not activated. The second loss corresponding to any target network model is determined based on the prediction results and annotation results corresponding to any target network model; The first task model is trained based on the first loss and second loss corresponding to each target network model to obtain a multi-task model.

2. The method according to claim 1, characterized in that, Each feature processing network includes multiple processing sub-networks, and each processing sub-network includes network parameters for multiple channels; the feature processing networks at a given location are merged to obtain a first merged network, including: For any sub-processing network included in any feature processing network, select the network parameters of a specified channel from the network parameters of each channel included in the sub-processing network. For any of the feature processing sub-networks at a given location, a merged sub-network is constructed based on the network parameters of the specified channel corresponding to any of the multiple feature processing sub-networks.

3. The method according to claim 2, characterized in that, Selecting network parameters for a specified channel from the network parameters of each channel included in any of the processing sub-networks includes: The network parameters of each channel in any of the processing sub-networks are normalized to obtain the normalized network parameters of each channel. Based on the normalized network parameters of each channel, the information entropy of each channel is determined; From the various channels, determine a specified channel whose information entropy is not less than the index threshold, and obtain the network parameters of the specified channel.

4. The method according to claim 1, characterized in that, The construction of the first task model based on each first merging network includes: For any first merging network, a first reconstructed network is constructed based on the first merging network and the initial control network, wherein the initial control network is used to control the output of the first reconstructed network; Based on each first reconstruction network, construct the first task model.

5. The method according to claim 4, characterized in that, The various first reconstruction networks are arranged in multiple layers. The step of processing the sample media data of any one target network model using the first task model to obtain the prediction result corresponding to that target network model includes: For the first layer first reconstruction network, feature extraction is performed on the sample media data of any target network model through the first layer first reconstruction network to obtain the media features output by the first layer first reconstruction network; For a non-first-layer first reconstruction network, feature extraction is performed on the media features output by the first reconstruction network above the non-first-layer first reconstruction network to obtain the media features output by the non-first-layer first reconstruction network. Based on the media features output by the last layer of the first reconstruction network, the prediction result corresponding to any target network model is determined.

6. The method according to claim 5, characterized in that, The first-layer first reconstruction network includes multiple first reconstruction networks, and the first merging network in the first reconstruction network includes an attention layer, a hidden layer, and an output layer; the feature extraction of sample media data of any target network model through the first-layer first reconstruction network to obtain the media features output by the first-layer first reconstruction network includes: For any of the first reconstruction networks included in the first layer of the first reconstruction network, the attention layer performs attention processing on the sample media data of the target network model to obtain the attention processing result; the hidden layer performs feature crossing on the attention processing result to obtain the feature crossing result; the initial control network controls the feature crossing result based on the attention processing result to obtain the control result; and the output layer determines the output result based on the control result. Based on the output results of each first reconstruction network included in the first layer first reconstruction network, the media characteristics output by the first layer first reconstruction network are determined.

7. The method according to claim 6, characterized in that, The initial control network includes a mapping network and an activation network, and the feature cross-results include sub-results from multiple channels; The step of regulating the feature cross result through an initial regulation network based on the attention processing result to obtain the regulation result includes: The attention processing results are mapped using the mapping network to obtain the mapping results for each channel; The activation network is used to activate the mapping results of each channel to obtain the activation results of each channel. The activation results of each channel are used to characterize the degree of activation of the channel. The control result is determined based on the activation results of each channel and the sub-results of the multiple channels.

8. The method according to claim 4, characterized in that, The method further includes: Train each initial control network in the first task model to obtain each first control network; The first merging network and the first control network are trained alternately to obtain a multi-task model.

9. A method for processing media data, characterized in that, The method includes: Acquire target media data; The target media data is processed by a multi-task model to obtain the results of each task. The multi-task model is trained by the training method of the multi-task model according to any one of claims 1 to 8.

10. The method according to claim 9, characterized in that, The multi-task model includes a multi-layer target reconstruction network; the process of performing at least one task on the target media data through the multi-task model to obtain the results of each task includes: For the first-layer target reconstruction network, feature extraction is performed on the target media data through the first-layer target reconstruction network to obtain the media features output by the first-layer target reconstruction network; For non-first-layer target reconstruction networks, feature extraction is performed on the media features output by the target reconstruction network above the non-first-layer target reconstruction network to obtain the media features output by the non-first-layer target reconstruction network. Based on the media features output by the network from the last layer target reconstruction, the processing results of each task are determined.

11. A training device for a multi-task model, characterized in that, The device includes: The acquisition module is used to acquire at least two target network models. Each target network model includes multiple feature processing networks, each feature processing network includes a multi-layer processing sub-network, and each layer processing sub-network includes network parameters for multiple channels. The topological connection structure formed by the multiple feature processing networks included in each target network model is the same. Each target network model is used to perform a task processing on target media data, and the target media data includes at least one of text, image, audio, and video. The merging module is used to merge feature processing networks located at the same position in the topological connection structure of the at least two target network models, and merge the feature processing networks at one position to obtain a first merged network; wherein, the first merged network includes multi-layer merged sub-networks, and each merged sub-network includes network parameters of a specified channel in any layer of the feature processing sub-network located at the same position whose information entropy is not less than an index threshold. The building module is used to construct the first task model based on each first merging network; The training module is used to acquire sample media data and annotation results for each target network model. The sample media data includes at least one of text, image, audio, and video. The sample media data of any target network model is used to train the target network model, and the annotation results of the target network model are used to annotate the processing results obtained by performing task processing on the sample media data. For the sample media data of any target network model, the first task model is used to perform corresponding task processing on the sample media data of the target network model to obtain the prediction result corresponding to the target network model. A first loss corresponding to the target network model is determined based on the prediction activation result and standard activation result of each specified channel in the first task model. The prediction activation result of the specified channel represents the activation degree of the specified channel, and the standard activation result of the specified channel represents the activation of the specified channel corresponding to the target network model or the non-activation of the specified channels corresponding to other target network models. A second loss corresponding to the target network model is determined based on the prediction result and annotation results. The first task model is trained based on the first loss and second loss corresponding to each target network model to obtain a multi-task model.

12. The apparatus according to claim 11, characterized in that, Each feature processing network includes multiple processing sub-networks, and each processing sub-network includes network parameters for multiple channels. The merging module is used to select network parameters of a specified channel from the network parameters of each channel in any sub-processing layer of any feature processing network; and to construct a merged sub-network based on the network parameters of the specified channel corresponding to multiple sub-processing layers of each feature processing network at a given location.

13. The apparatus according to claim 12, characterized in that, The merging module is used to normalize the network parameters of each channel in any layer of the processing sub-network to obtain the normalized network parameters of each channel; and to determine the information entropy of each channel based on the normalized network parameters of each channel. From the various channels, determine a specified channel whose information entropy is not less than the index threshold, and obtain the network parameters of the specified channel.

14. The apparatus according to claim 11, characterized in that, The construction module is used to construct a first reconstruction network based on any first merging network and an initial control network for any first merging network, wherein the initial control network is used to control the output of the first reconstruction network; and to construct a first task model based on each first reconstruction network.

15. The apparatus according to claim 14, characterized in that, The various first reconstruction networks are arranged in multiple layers; The training module is used to extract features from the sample media data of any target network model through the first reconstruction network of the first layer to obtain the media features output by the first reconstruction network of the first layer; and to extract features from the media features output by the first reconstruction network of the previous layer of the non-first reconstruction network through the non-first reconstruction network to obtain the media features output by the non-first reconstruction network. Based on the media features output by the last layer of the first reconstruction network, the prediction result corresponding to any target network model is determined.

16. The apparatus according to claim 15, characterized in that, The first layer first reconstruction network includes multiple first reconstruction networks, and the first merging network in the first reconstruction network includes an attention layer, a hidden layer and an output layer; The training module is used to perform attention processing on the sample media data of the target network model through the attention layer for any of the first reconstruction networks included in the first layer first reconstruction network, and obtain the attention processing result; The attention processing result is subjected to feature crossing through the hidden layer to obtain the feature crossing result; The feature cross result is adjusted based on the attention processing result through the initial adjustment network to obtain the adjustment result; the output result is determined by the output layer based on the adjustment result. Based on the output results of each first reconstruction network included in the first layer first reconstruction network, the media characteristics output by the first layer first reconstruction network are determined.

17. The apparatus according to claim 16, characterized in that, The initial control network includes a mapping network and an activation network, and the feature cross-results include sub-results from multiple channels; The training module is used to map the attention processing results through the mapping network to obtain the mapping results for each channel; The activation network is used to activate the mapping results of each channel to obtain the activation results of each channel. The activation results of each channel are used to characterize the degree of activation of the channel. The control result is determined based on the activation results of each channel and the sub-results of the multiple channels.

18. The apparatus according to claim 14, characterized in that, The training module is used to train each initial control network in the first task model to obtain each first control network; and to alternately train each first merging network and each first control network to obtain a multi-task model.

19. A media data processing apparatus, characterized in that, The device includes: The acquisition module is used to acquire target media data; The task processing module is used to perform at least one task processing on the target media data through a multi-task model to obtain the results of each task processing, wherein the multi-task model is trained according to the training method of the multi-task model according to any one of claims 1 to 8.

20. The apparatus according to claim 19, characterized in that, The multi-task model includes a multi-layer target reconstruction network; The task processing module is used to extract features from the target media data through the first-layer target reconstruction network to obtain the media features output by the first-layer target reconstruction network; and to extract features from the media features output by the target reconstruction network above the non-first-layer target reconstruction network to obtain the media features output by the non-first-layer target reconstruction network. Based on the media features output by the network from the last layer target reconstruction, the processing results of each task are determined.

21. An electronic device, characterized in that, The electronic device includes a processor and a memory, the memory storing at least one computer program, the at least one computer program being loaded and executed by the processor to enable the electronic device to perform the method as described in any one of claims 1 to 10.

22. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to enable the electronic device to perform the method as described in any one of claims 1 to 10.

23. A computer program product, characterized in that, The computer program product stores at least one computer program, which is loaded and executed by a processor to enable the electronic device to perform the method as described in any one of claims 1 to 10.

Images