Identification method and device, electronic equipment and storage medium

Abstract

The application provides a recognition method and device, electronic equipment and storage medium. The method comprises: acquiring multimedia data to be recognized; determining an emotion recognition result and / or an intention recognition result of the multimedia data based on a recognition model; the recognition model is obtained based on sample multimedia data, emotion labels and intention labels of the sample multimedia data, and multi-task learning including emotion recognition and intention recognition is performed. The method, device, electronic equipment and storage medium provided by the application apply the recognition model obtained by multi-task learning including emotion recognition and intention recognition to perform emotion recognition and / or intention recognition, so that information related to emotion and intention in the multimedia data can be fully applied, and the result accuracy of emotion recognition and intention recognition is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of machine learning, and in particular to a recognition method and device, electronic equipment and a storage medium. BACKGROUND

[0002] With the development of artificial intelligence, in recent years, intent recognition technology and emotion recognition technology have developed rapidly. For example, in the driving field, the driver's intent changes and emotional fluctuations can be recognized, thereby reducing the occurrence of accidents.

[0003] In the prior art, most of the intent recognition models and emotion recognition models are distinguished, and two independent models are applied to realize intent recognition and emotion recognition. The above scheme can obtain intent recognition results and emotion recognition results, but the accuracy of the results of intent recognition and emotion recognition is low, which affects the reliability of subsequent applications. SUMMARY

[0004] The present application provides a recognition method, device, electronic equipment and storage medium to solve the problem of low accuracy of intent recognition and emotion recognition results caused by applying two independent models to realize intent recognition and emotion recognition in the prior art.

[0005] The present application provides a recognition method, comprising:

[0006] Obtaining multimedia data to be recognized;

[0007] Determining the emotion recognition result and / or the intent recognition result of the multimedia data based on a recognition model;

[0008] The recognition model is obtained based on sample multimedia data, and emotion labels and intent labels of the sample multimedia data, and performs multi-task learning including emotion recognition and intent recognition.

[0009] According to the recognition method provided by the present application, the recognition model obtaining step comprises:

[0010] Obtaining an initial model, the initial model comprising an initial encoding layer, and an initial emotion recognition branch and an initial intent recognition branch connected to the initial encoding layer respectively;

[0011] Based on the initial encoding layer, the sample multimedia data is feature-encoded to obtain sample multimedia features of the sample multimedia data;

[0012] Based on the initial emotion recognition branch, the sample multimedia features are emotion-recognized to obtain emotion prediction results of the sample multimedia data;

[0013] Based on the initial intention recognition branch, the sample multimedia feature is subjected to intention recognition to obtain an intention prediction result of the sample multimedia data.

[0014] Based on the emotion prediction result, the intention prediction result, and emotion labels and intention labels of the sample multimedia data, the initial model is subjected to parameter iteration to obtain the recognition model.

[0015] According to the recognition method provided by the application, the obtaining step of the initial encoding layer comprises:

[0016] Based on the pre-training encoding model of multiple modalities, the initial encoding layer is constructed, and the multiple modalities comprise at least two of an audio modality, a text modality, and a video modality.

[0017] According to the recognition method provided by the application, the pre-training encoding model of the audio modality is trained based on sample speech and transcribed text;

[0018] The pre-training encoding model of the text modality is obtained by self-supervised training of a language model based on sample text;

[0019] The pre-training encoding model of the video modality is obtained by self-supervised training based on sample images.

[0020] According to the recognition method provided by the application, the multimedia data comprises audio-video data;

[0021] The emotion recognition result and / or the intention recognition result of the multimedia data are determined based on the recognition model, comprising:

[0022] Based on the feature extraction layer in the recognition model, speech features, text features, and video features of the audio-video data are extracted and fused to obtain multimedia features of the audio-video data;

[0023] Based on the emotion recognition branch in the recognition model, the multimedia features are subjected to emotion recognition to obtain an emotion recognition result of the audio-video data, and / or, based on the intention recognition branch in the recognition model, the multimedia features are subjected to intention recognition to obtain an intention recognition result of the audio-video data.

[0024] According to the recognition method provided by the application, based on the feature extraction layer in the recognition model, speech features, text features, and video features of the audio-video data are extracted and fused to obtain multimedia features of the audio-video data, comprising:

[0025] Based on the speech encoding layer in the feature extraction layer, the audio of the audio-video data is subjected to speech encoding to obtain the speech features;

[0026] decoding the speech feature based on a speech decoding layer in the feature extraction layer to obtain a transcription text of the audio;

[0027] encoding the transcription text based on a semantic encoding layer in the feature extraction layer to obtain the text feature;

[0028] encoding the video in the audio-video data based on a video encoding layer in the feature extraction layer to obtain the video feature;

[0029] fusing the speech feature, the text feature and the video feature based on a fusion encoding layer in the feature extraction layer to obtain the multimedia feature.

[0030] According to the recognition method provided by the application, the fusion encoding layer in the feature extraction layer is used to fuse the speech feature, the text feature and the video feature to obtain the multimedia feature, which comprises the following steps:

[0031] projecting the speech feature, the text feature and the video feature to the same feature dimension based on a projection layer in the fusion encoding layer to obtain projected speech feature, projected text feature and projected video feature;

[0032] performing attention transformation on the sum of the projected speech feature, the projected text feature and the projected video feature based on an attention layer in the fusion encoding layer to obtain an attention feature;

[0033] applying the sum of the projected speech feature, the projected text feature and the projected video feature and the attention feature to a residual encoding layer in the fusion encoding layer to determine the multimedia feature.

[0034] The application further provides a recognition device, which comprises:

[0035] an acquisition unit configured to acquire multimedia data to be recognized;

[0036] a determination unit configured to determine an emotion recognition result and / or an intention recognition result of the multimedia data based on a recognition model;

[0037] The recognition model is obtained based on sample multimedia data and emotion labels and intention labels of the sample multimedia data through multi-task learning including emotion recognition and intention recognition.

[0038] The application further provides an electronic device, which comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and the processor implements the recognition method according to any one of the above embodiments when executing the program.

[0039] The application further provides a non-transitory computer-readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the identification method.

[0040] The application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the identification method.

[0041] The identification method, device, electronic equipment and storage medium provided by the application apply the identification model obtained by multi-task learning including emotion recognition and intention recognition to emotion recognition and / or intention recognition, so that information related to emotions and intentions in multimedia data can be fully applied, and the result accuracy of emotion recognition and intention recognition is improved. Moreover, one identification model can be used to simultaneously realize intention recognition and emotion recognition, compared with a scheme in which two independent models are used to respectively realize intention recognition and emotion recognition, the model training cost is reduced, and the operation efficiency when intention recognition and emotion recognition are performed in parallel is improved. BRIEF DESCRIPTION OF DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are some embodiments of the application, and for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0043] Figure 1 is one of the flowcharts of the identification method provided by the application;

[0044] Figure 2 is the flowchart of the obtaining step of the identification model provided by the application;

[0045] Figure 3 is the flowchart of step 120 in the identification method provided by the application;

[0046] Figure 4 is the flowchart of the emotion recognition and / or intention recognition on the multimedia features provided by the application;

[0047] Figure 5 is the structural diagram of the speech coding layer provided by the application;

[0048] Figure 6 is the structural diagram of the video coding layer provided by the application;

[0049] Figure 7 is the flowchart of obtaining the multimedia features provided by the application;

[0050] Figure 8is a structural schematic diagram of a fusion coding layer provided by the present application;

[0051] Figure 9 is a second flow schematic diagram of the identification method provided by the present application;

[0052] Figure 10 is a structural schematic diagram of the identification device provided by the present application;

[0053] Figure 11 is a structural schematic diagram of the electronic device provided by the present application. DETAILED DESCRIPTION

[0054] In order to make the objectives, technical solutions and advantages of the present application clearer, the technical solutions in the present application will be described clearly and completely below with reference to the drawings in the present application. Obviously, the described embodiments are some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0055] The present application provides an identification method, Figure 1 is one of the flow schematic diagrams of the identification method provided by the present application, as Figure 1 shown, the method comprises:

[0056] Step 110, obtaining multimedia data to be identified.

[0057] Specifically, the multimedia data to be identified is the multimedia data that needs to be identified for emotion and / or intention, and the multimedia data can include at least one of voice data, text data and video data, i.e. the multimedia data can only include voice data, can only include video data, or can include both voice data and video data, and the present application does not make specific limitation on this.

[0058] Among them, the voice data can be obtained by a sound pickup device, which can be a smart phone, a tablet computer, or a smart appliance such as a sound box, a television and an air conditioner, etc. After the sound pickup device obtains voice data through a microphone array, it can also amplify and denoise the voice data. The text data can be obtained by transcribing the voice data, or by crawling on a webpage. The video data can be obtained by pre-acquiring through a video acquisition device, or by real-time recording, and the present application does not make specific limitation on this.

[0059] Step 120, determining the emotion recognition result and / or the intention recognition result of the multimedia data based on an identification model;

[0060] The recognition model is obtained by performing multi-task learning including emotion recognition and intention recognition based on sample multimedia data, and emotion labels and intention labels of the sample multimedia data.

[0061] Specifically, in most scenarios, there is a strong correlation between the intention and emotion of a human. For example, "a man and a woman are arguing, and the woman turns her head to look at a vase beside her", from which it can be predicted that the emotion recognition result of the woman is likely to be "angry", and the intention recognition result of the woman is likely to be "throwing the vase". It can be understood that a human may have the intention of "throwing the vase" in the emotion of "angry", and the human is usually in the emotion of "angry" when having the intention of "throwing the vase", so it can be seen that there is a correlation between intention recognition and emotion recognition.

[0062] Accordingly, in the training of the recognition model, a multi-task learning mode including emotion recognition and intention recognition is adopted. Here, the multi-task learning at least includes an emotion recognition task and an intention recognition task. Specifically, in the multi-task learning process, sample multimedia data carrying two types of labels, emotion labels and intention labels, can be collected first, and then the sample multimedia data is taken as the input of an initial model to obtain the results of emotion recognition and intention recognition of the sample multimedia data by the initial model, and the initial model is iterated based on the difference between the emotion recognition result of the sample multimedia data and the emotion label, and the difference between the intention recognition result of the sample multimedia data and the intention label, so as to obtain the recognition model.

[0063] It can be understood that in the multi-task learning process of the recognition model, emotion recognition and intention recognition are parallel, that is, the recognition model can learn the features of the multimedia data common to emotion recognition and intention recognition in the multi-task learning process. The recognition model trained in this way can extract the features of the multimedia data common to emotion recognition and intention recognition and put them into subsequent applications, whether it is used for emotion recognition alone, or used for intention recognition alone, or used for both emotion recognition and intention recognition. Therefore, the emotion recognition result of the multimedia data output by the recognition model not only refers to the related information of the human emotion contained in the multimedia data itself, but also can refer to the related information of the human intention contained in the multimedia data itself; similarly, the intention recognition result of the multimedia data output by the recognition model not only refers to the related information of the human intention contained in the multimedia data itself, but also can refer to the related information of the human emotion contained in the multimedia data itself. The emotion recognition result here can be any one of happy, angry, sad, surprised, disgusted, frightened and neutral, and the intention recognition result here can be double-hand clapping, throwing things, shopping, running, walking, etc., which are not limited by the embodiments of the present application.

[0064] That is, the recognition model obtained through multi-task learning including emotion recognition and intention recognition can fully apply the correlation between emotion recognition and intention recognition in the process of emotion recognition and / or intention recognition to improve the accuracy of the emotion recognition result and / or the intention recognition result.

[0065] The method provided by the embodiments of the present application applies the recognition model obtained through multi-task learning including emotion recognition and intention recognition to perform emotion recognition and / or intention recognition, so that information related to emotion and intention in multimedia data can be fully applied, and the result accuracy of emotion recognition and intention recognition is improved. Moreover, one recognition model can be used to simultaneously realize intention recognition and emotion recognition, which reduces the model training cost and improves the operation efficiency when intention recognition and emotion recognition are performed in parallel, compared with a scheme in which two independent models are respectively used to realize intention recognition and emotion recognition.

[0066] Based on the above embodiments, Figure 2 is a flowchart of the obtaining step of the recognition model provided by the present application, as Figure 2 shown, the obtaining step of the recognition model includes:

[0067] Step 210, obtaining an initial model, the initial model including an initial encoding layer, and an initial emotion recognition branch and an initial intention recognition branch respectively connected with the initial encoding layer;

[0068] Step 220, based on the initial encoding layer, performing feature encoding on the sample multimedia data to obtain sample multimedia features of the sample multimedia data;

[0069] Step 230, based on the initial emotion recognition branch, performing emotion recognition on the sample multimedia features to obtain an emotion prediction result of the sample multimedia data;

[0070] based on the initial intention recognition branch, performing intention recognition on the sample multimedia features to obtain an intention prediction result of the sample multimedia data;

[0071] Step 240, based on the emotion prediction result, the intention prediction result, and emotion labels and intention labels of the sample multimedia data, performing parameter iteration on the initial model to obtain the recognition model.

[0072] Here, the initial model includes an initial encoding layer, and an initial emotion recognition branch and an initial intention recognition branch respectively connected with the initial encoding layer, that is, the initial emotion recognition branch and the initial intention recognition branch share the initial encoding layer. The parameters of the initial model here can be randomly generated or pre-set.

[0073] During the training of the recognition model, sample multimedia data can be input into the initial encoding layer. The initial encoding layer encodes the features of the sample multimedia data, obtaining and outputting the sample multimedia features. It can be understood that the output of the initial encoding layer is the input to both the initial emotion recognition branch and the initial intent recognition branch; that is, the input features of the initial emotion recognition branch and the initial intent recognition branch are consistent. In other words, the sample multimedia features output by the initial encoding layer include both emotion-related and intent-related information from the sample multimedia data.

[0074] Subsequently, the multimedia features of the sample can be input into the initial emotion recognition branch, which performs emotion recognition on the multimedia features of the sample and outputs the emotion prediction result of the multimedia data. The initial emotion recognition branch here may include a linear layer and an activation layer. The linear layer can use a linear function, and the activation layer can use the GELU (Gaussian Error Linear Unit) activation function, the Softmax activation function, or the ReLU (Rectified Linear Units) activation function. This embodiment of the invention does not specifically limit the specific activation function used.

[0075] Simultaneously, the sample multimedia features can be input into the initial intent recognition branch, which then performs intent recognition on the sample multimedia features to obtain and output the intent prediction result of the sample multimedia data. This initial intent recognition branch can include a linear layer and an activation layer. The linear layer can use a Linear function, and the activation layer can use the GELU activation function, the Softmax activation function, or the ReLU activation function; this embodiment of the invention does not specifically limit the specific activation function used. Furthermore, the structures of the initial emotion recognition branch and the initial intent recognition branch can be the same or different.

[0076] After obtaining the emotion prediction results from the initial emotion recognition branch and the intent prediction results from the initial intent recognition branch, a first loss function can be determined based on the emotion prediction results and the emotion labels of the sample multimedia data. This first loss function reflects the difference between the emotion prediction results and the emotion labels. Similarly, a second loss function can be determined based on the intent prediction results and the intent labels of the sample multimedia data. This second loss function reflects the difference between the intent prediction results and the intent labels. Then, the total loss can be determined based on the first and second loss functions, and the initial model's parameters can be iterated based on the total loss to obtain the recognition model. During this process, the parameter iteration of the initial encoding layer in the initial model simultaneously learns the features required for both emotion recognition and intent recognition. This enables the encoding layer used in the trained recognition model to output the features required for both emotion recognition and intent recognition.

[0077] The method provided in this invention uses the same initial encoding layer for emotion recognition and intent recognition in the initial model. This allows subsequent applications, whether for emotion recognition or intent recognition, to reference the features required for the other task, thereby improving the accuracy of recognition.

[0078] Most related technologies utilize single-modal data for emotion recognition or intent recognition. For example, intent recognition is based on speech modality data, emotion recognition is based on video modality data, or emotion recognition is based on text modality data.

[0079] The aforementioned recognition scheme based on single-modal data has low accuracy, and it is no longer sufficient to meet the needs of complex real-world scenarios. To address these issues, this invention improves the initial encoding layer of the initial model.

[0080] Based on the above embodiments, the initial coding layer acquisition step in the initial model includes:

[0081] The initial coding layer is constructed based on a pre-trained coding model with multiple modalities, wherein the multiple modalities include at least two of the following: audio modality, text modality, and video modality.

[0082] Specifically, compared to single-modal data recognition schemes in related technologies, the initial encoding layer in this embodiment of the invention is constructed based on a pre-trained encoding model with at least two modalities. That is, the initial encoding layer in this embodiment can achieve feature encoding for sample multimedia data including at least two modalities. Here, "at least two modalities" can include audio and text modalities, text and video modalities, or audio, text, and video modalities; this embodiment of the invention does not specifically limit this.

[0083] Specifically, the initial encoding layer construction process utilizes pre-trained encoding models for each modality across multiple modalities. Here, the pre-trained encoding model for any given modality is a pre-trained model capable of encoding data specific to that modality. For example, the pre-trained encoding model for the text modality could be a BERT (Bidirectional Encoder Representations from Transformer) model. By constructing the initial encoding layer using pre-trained encoding models from multiple modalities, good feature extraction results can be obtained with only a small amount of training data during subsequent parameter iterations. This significantly reduces the training cost of the encoding layer in the recognition model and improves its training efficiency.

[0084] Furthermore, the encoding layer in the recognition model trained in this way also has the ability to encode features of multimedia data that includes multiple modalities. That is, the recognition model itself can be applied to emotion recognition and / or intent recognition of multimedia data that includes multiple modalities. Moreover, in the face of complex real-world scenarios, the recognition model can perform emotion recognition and / or intent recognition of multimedia data with multiple modalities, thus meeting the needs of real-world scenarios.

[0085] Based on the above embodiments, the pre-trained encoding model of the audio modality is obtained by training on sample speech and transcribed text;

[0086] The pre-trained encoding model of the text modality is obtained by self-supervised training of the language model based on sample text;

[0087] The pre-trained encoding model for the video modality is obtained through self-supervised training based on sample images.

[0088] Specifically, the pre-trained encoding model for audio modalities is trained based on the differences between the speech features obtained by encoding the sample speech and the transcribed text obtained by decoding the encoded speech features. It can be understood that the smaller the difference between the sample speech and the transcribed text, the better the match between the sample speech and the transcribed text, and the better the training effect of the pre-trained encoding model for audio modalities.

[0089] The pre-trained encoding model for the text modality is obtained by self-supervised learning of the language model based on sample text. Here, self-supervised training of the speech model refers to using proxy tasks to extract the representational characteristics of the sample text itself as supervisory information for unlabeled sample text data, thereby improving the model's feature extraction ability. For example, some words in the sample text are randomly masked, and the masked text is used as input, while the original unmasked text is used as the label. The model is then trained to predict the words in the masked portion. For instance, if the sample text is "A quick brown fox jumps over the lazy dog," and the randomly masked text is "A quick [MASK] fox jumps over the [MASK] dog," the model is expected to predict the words in the [MASK] portion, and the encoded part of the model is used as the pre-trained encoding model for the text modality.

[0090] The pre-trained encoding model for video modalities is obtained through self-supervised training based on sample images. Here, self-supervised training based on sample images refers to using auxiliary tasks to extract the inherent representational features of the sample images as supervisory information for unlabeled sample image data, thereby improving the model's feature extraction capabilities. For example, a portion of a face in a sample image is randomly masked. The masked image is used as input, and the original unmasked image is used as the label to train the model's ability to predict the masked portion of the face. For instance, if the sample image is a face image and the nose is randomly masked, the model is expected to predict the nose, and the encoded portion of this model is used as the pre-trained encoding model for the video modality.

[0091] The method provided in this invention pre-trains the coding model on samples of multiple modalities, thereby enabling the recognition of input audio modalities, text modalities, and video modalities, improving the model's generalization ability. Furthermore, the pre-trained coding model can better learn the features of each modality, thus improving the model's coding accuracy.

[0092] Based on the above embodiments, Figure 3 This is a flowchart illustrating step 120 of the identification method provided by the present invention, as shown below. Figure 3 As shown, the multimedia data includes audio and video data;

[0093] Step 120 includes:

[0094] Step 121: Based on the feature extraction layer in the recognition model, extract and fuse the speech features, text features and video features of the audio and video data to obtain the multimedia features of the audio and video data;

[0095] Step 122: Based on the emotion recognition branch in the recognition model, perform emotion recognition on the multimedia features to obtain the emotion recognition result of the audio and video data, and / or, based on the intent recognition branch in the recognition model, perform intent recognition on the multimedia features to obtain the intent recognition result of the audio and video data.

[0096] Specifically, audio and video data includes audio data and video data, where video data is an image sequence. After acquiring the audio and video data, speech features, text features, and video features can be extracted from the audio and video data based on the feature extraction layer in the recognition model. Here, speech features refer to the features corresponding to the audio data extracted from the audio and video data based on the feature extraction layer in the recognition model; text features refer to the features corresponding to the text data extracted after transing the audio data from the audio and video data into text data based on the feature extraction layer in the recognition model; and video features refer to the features corresponding to the video data extracted from the audio and video data based on the feature extraction layer in the recognition model.

[0097] Subsequently, the speech features, text features, and video features of the audio and video data are fused based on the feature extraction layer to obtain the multimedia features of the audio and video data. The fusion of speech features, text features, and video features here can be achieved by concatenating the speech features, text features, and video features, or by using an attention mechanism to weight the speech features, text features, and video features before concatenation. This embodiment of the invention does not specifically limit the method used.

[0098] It is understandable that the multimedia features here are features that integrate speech, text, and video features; that is, the multimedia features here are features from three modalities: speech, text, and video. Therefore, emotion recognition can be performed on the multimedia features based on the emotion recognition branch of the recognition model to obtain the emotion recognition results for the audio and video data, and / or, intent recognition can be performed on the multimedia features based on the intent recognition branch of the recognition model to obtain the intent recognition results for the audio and video data.

[0099] Based on the above embodiments, Figure 4 This is a flowchart illustrating the process of emotion recognition and / or intent recognition based on multimedia features provided by the present invention, such as... Figure 4 As shown, the emotion recognition branch here can include two linear layers and one activation layer. The linear layers can use the Linear function, and the activation layer can use the Softmax activation function. Similarly, the intent recognition branch here can include two linear layers and one activation layer. The linear layers can use the Linear function, and the activation layer can use the Softmax activation function.

[0100] Based on the above embodiments, step 121 includes:

[0101] Based on the speech coding layer in the feature extraction layer, the audio of the audio and video data is speech encoded to obtain the speech features;

[0102] Based on the speech decoding layer in the feature extraction layer, the speech features are decoded to obtain the transcribed text of the audio.

[0103] Based on the semantic encoding layer in the feature extraction layer, the transcribed text is semantically encoded to obtain the text features;

[0104] Based on the video encoding layer in the feature extraction layer, the video in the audio and video data is encoded to obtain the video features;

[0105] Based on the fusion coding layer in the feature extraction layer, the speech features, text features, and video features are fused to obtain the multimedia features.

[0106] Specifically, the feature extraction layer in the recognition model includes a speech coding layer, a speech decoding layer, a semantic coding layer, a video coding layer, and a fusion coding layer. Among these, the speech coding layer in the feature extraction layer can be used to encode the audio of the audio and video data to obtain speech features. Figure 5 This is a schematic diagram of the speech coding layer provided by the present invention, as shown below. Figure 5 As shown, for example, the speech coding layer can be a CTC (Connectionist Temporal Classification) end-to-end speech coding model. This speech coding model can include convolutional layers, a Conformer (CNN+Transformers) network, linear layers, and activation layers. The input of the convolutional layer is the input of the speech coding layer, the output of the convolutional layer is the input of the Conformer network, the output of the Conformer network is the input of the linear layers and activation layers, and the output of the linear layers and activation layers is the output of the speech coding layer.

[0107] The convolutional layers here can be Deep-ConvNet or Fully Convolutional Networks (FCN). The activation layers here can be GELU activation function, Softmax activation function, or ReLU activation function. This embodiment of the invention does not make specific limitations on these.

[0108] After obtaining the speech features, the speech features can be decoded based on the speech decoding layer in the feature extraction layer to obtain the transcribed text of the audio. The speech decoding layer here uses a greedy strategy to decode the speech features, and the transcribed text of the audio is the text corresponding to the speech features.

[0109] After obtaining the transcribed text, semantic encoding can be performed on the transcribed text based on the semantic encoding layer in the feature extraction layer to obtain text features. This semantic encoding layer can include the BERT model, and a special [CLS] identifier is added to the beginning of each sentence in the transcribed text to represent the semantic information of the entire sentence. The [CLS] identifier here is a weighted average of each word after the self-attention mechanism in the BERT model.

[0110] Since audio and video data also contains video data, video encoding can be performed on the video data within the feature extraction layer to obtain video features. Here, the video encoding layer is used to extract the speaker's facial expressions and movements. Figure 6 This is a schematic diagram of the video coding layer provided by the present invention, as shown below. Figure 6 As shown, the video coding layer may include a 3D convolutional network and a ResNet18 network. The input of the 3D convolutional network is the input of the video coding layer, the output of the 3D convolutional network is the input of the ResNet18 network, and the output of the ResNet18 network is the output of the video coding layer.

[0111] After obtaining speech, text, and video features, the fusion coding layer in the feature extraction layer can be used to fuse these features to obtain multimedia features. This fusion coding layer can combine features from multiple modalities to obtain multimedia features.

[0112] Based on the above embodiments, Figure 7 This is a schematic diagram of the process for obtaining multimedia features provided by the present invention, such as... Figure 7 As shown, the multimedia features obtained by fusing the speech features, text features, and video features based on the fusion coding layer in the feature extraction layer include:

[0113] Step 710: Based on the projection layer in the fusion coding layer, the speech features, text features and video features are projected onto the same feature dimension to obtain the projected speech features, projected text features and projected video features.

[0114] Step 720: Based on the attention layer in the fusion coding layer, perform attention transformation on the sum of the projected speech features, projected text features, and projected video features to obtain attention features;

[0115] Step 730: Based on the residual coding layer in the fusion coding layer, the multimedia features are determined by applying the sum of the projected speech features, projected text features, and projected video features, as well as the attention features.

[0116] Specifically, Figure 8 This is a schematic diagram of the structure of the fusion coding layer provided by the present invention, as shown below. Figure 8 As shown, the fusion coding layer includes a projection layer, an attention layer, and a residual coding layer. The input to the fusion coding layer is also the input to the projection layer. The projection layer can use a linear function to project speech features, text features, and video features onto the same feature dimension, resulting in projected speech features, projected text features, and projected video features. It can be understood that the existence of the projection layer is to project speech features, text features, and video features onto the same feature dimension, facilitating subsequent attention transformation and residual coding of the speech features, text features, and video features.

[0117] Here, the same feature dimension can be any one of the speech feature dimension, text feature dimension, and video feature dimension, or it can be a feature dimension other than the three feature dimensions mentioned above. It is understood that, in order to obtain projected speech features, projected text features, and projected video features, the speech features, text features, and video features can be projected onto the speech feature dimension. In this case, the speech features projected onto the speech feature dimension are recorded as projected speech features, the text features projected onto the speech feature dimension are recorded as projected text features, and the video features projected onto the speech feature dimension are recorded as projected video features. Alternatively, the speech features, text features, and video features can be projected onto the text feature dimension. In this case, the speech features projected onto the text feature dimension are recorded as projected speech features, the text features projected onto the text feature dimension are recorded as projected text features, and the video features projected onto the text feature dimension are recorded as projected video features. Furthermore, the speech features, text features, and video features can be projected onto the video feature dimension. In this case, the speech features projected onto the video feature dimension are recorded as projected speech features, the text features projected onto the video feature dimension are recorded as projected text features, and the video features projected onto the video feature dimension are recorded as projected video features. This embodiment of the invention does not specifically limit the specific implementation of these methods.

[0118] The projection layer design makes projections for language features, text features, and video features more flexible and easier to match the characteristics of language features, text features, and video features, which helps to improve the accuracy of emotion recognition and intent recognition.

[0119] The projected speech features, projected text features, and projected video features are weighted and fused. This weighted fused feature is then input into an attention layer, which can employ a multi-head self-attention mechanism. This attention layer performs an attention transformation on the sum of the projected speech, text, and video features to obtain attention features. This attention layer learns the importance of language, text, and video features in emotion recognition and intent recognition, respectively, and weights them accordingly. This ensures that the output attention features highlight features effective for emotion and intent recognition while weakening features ineffective for these functions.

[0120] The output of the attention layer is the input of the residual coding layer. By fusing the residual coding layer within the fusion coding layer, the sum of projected speech features, projected text features, and projected video features, along with the attention features, can be applied to determine multimedia features. Specifically, the sum of the projected speech features, projected text features, and projected video features is weighted and summed with the attention features, or they are directly added together to obtain the multimedia features. Alternatively, further feature extraction can be performed on the weighted summed or added features to obtain the multimedia features. Here, further feature extraction can be implemented through a feed-forward network (FFN) layer. The resulting multimedia features, which fuse speech features, text features, and video features, can be used for emotion recognition and / or intent recognition.

[0121] Thus, by learning speech features, text features, and video features through the projection layer, attention layer, and residual coding layer, the fusion coding layer can learn effective information for subsequent emotion recognition and / or intent recognition.

[0122] Based on any of the above embodiments Figure 9 This is a second schematic diagram of the identification method provided by the present invention, as shown below. Figure 9 As shown, the method includes:

[0123] First, acquire the multimedia data to be identified. This multimedia data includes audio and video data, that is, both audio and video data.

[0124] Then, based on the feature extraction layer in the recognition model, the speech features, text features and video features of the audio and video data are extracted and fused. The feature extraction layer here includes a speech coding layer, a speech decoding layer, a semantic coding layer, a video coding layer and a fusion coding layer.

[0125] Specifically, the audio of the audio and video data can be encoded into speech based on the speech coding layer in the feature extraction layer to obtain speech features. For example, the speech coding layer can be a CTC end-to-end speech coding model. The CTC end-to-end speech coding model can include convolutional layers, a Conformer network, linear layers, and activation layers. The input of the convolutional layer is the input of the speech coding layer, the output of the convolutional layer is the input of the Conformer network, the output of the Conformer network is the input of the linear layer and the activation layer, and the output of the linear layer and the activation layer is the output of the speech coding layer.

[0126] The convolutional layers here can be deep fully convolutional networks or fully convolutional networks. The activation layers here can be GELU activation function, Softmax activation function, or ReLU activation function. This embodiment of the invention does not make specific limitations on these.

[0127] After obtaining the speech features, the speech features can be decoded based on the speech decoding layer in the feature extraction layer to obtain the transcribed text of the audio. Here, the speech decoding layer uses a greedy strategy to decode the speech features.

[0128] After obtaining the transcribed text, semantic encoding can be performed on the transcribed text based on the semantic encoding layer in the feature extraction layer to obtain text features. This semantic encoding layer can include the BERT model, and a special [CLS] identifier is added to the beginning of each sentence in the transcribed text to represent the semantic information of the entire sentence. The [CLS] identifier here is a weighted average of each word after the self-attention mechanism in the BERT model.

[0129] Since audio and video data also contains video data, video encoding can be performed on the video data within the feature extraction layer to obtain video features. Here, the video encoding layer is used to extract the speaker's facial expressions and movements. The video encoding layer can include a 3D convolutional network and a ResNet18 network. The input of the 3D convolutional network is the input of the video encoding layer, and the output of the 3D convolutional network is the input of the ResNet18 network. The output of the ResNet18 network is the output of the video encoding layer.

[0130] After obtaining speech features, text features, and video features, the fusion coding layer in the feature extraction layer can be used to fuse these features to obtain multimedia features. This fusion coding layer includes a projection layer, an attention layer, and a residual coding layer. The input to the fusion coding layer is also the input to the projection layer. The projection layer in the fusion coding layer projects the speech features, text features, and video features onto the same feature dimension, resulting in projected speech features, projected text features, and projected video features.

[0131] The output of the projection layer is the input of the attention layer. The attention layer can perform attention transformation on the sum of the projected speech features, projected text features, and projected video features to obtain attention features.

[0132] The output of the attention layer is the input of the residual coding layer. By fusing the residual coding layer within the coding layer, the sum of projected speech features, projected text features, and projected video features, along with the attention features, can be applied to determine multimedia features. These multimedia features, which integrate speech, text, and video features, can be used for emotion recognition and / or intent recognition, yielding emotion recognition and / or intent recognition results for the multimedia data.

[0133] The identification device provided by the present invention is described below. The identification device described below and the identification method described above can be referred to in correspondence.

[0134] Based on any of the above embodiments Figure 10 This is a schematic diagram of the structure of the identification device provided by the present invention, as shown below. Figure 10 As shown, the device includes:

[0135] Acquisition unit 1010 is used to acquire multimedia data to be identified;

[0136] The determining unit 1020 is used to determine the emotion recognition result and / or intent recognition result of the multimedia data based on the recognition model;

[0137] The recognition model is obtained by performing multi-task learning, including emotion recognition and intent recognition, based on sample multimedia data and the emotion and intent labels of the sample multimedia data.

[0138] The apparatus provided in this invention applies a recognition model obtained through multi-task learning, including emotion recognition and intent recognition, to perform emotion recognition and / or intent recognition. This allows for the full utilization of emotion and intent-related information in multimedia data, improving the accuracy of emotion and intent recognition results. Furthermore, a single recognition model can simultaneously achieve intent recognition and emotion recognition, reducing model training costs and improving computational efficiency when performing parallel intent and emotion recognition compared to using two independent models for each.

[0139] Based on any of the above embodiments, the step of obtaining the recognition model includes:

[0140] Obtain an initial model, which includes an initial encoding layer, and an initial emotion recognition branch and an initial intention recognition branch respectively connected to the initial encoding layer;

[0141] Based on the initial encoding layer, feature encoding is performed on the sample multimedia data to obtain the sample multimedia features of the sample multimedia data;

[0142] Based on the initial emotion recognition branch, emotion recognition is performed on the multimedia features of the sample to obtain the emotion prediction result of the multimedia data of the sample.

[0143] Based on the initial intent recognition branch, intent recognition is performed on the sample multimedia features to obtain the intent prediction result of the sample multimedia data;

[0144] Based on the emotion prediction results, the intention prediction results, and the emotion and intention labels of the sample multimedia data, the initial model is iterated to obtain the recognition model.

[0145] Based on any of the above embodiments, the step of obtaining the initial coding layer includes:

[0146] The initial coding layer is constructed based on a pre-trained coding model with multiple modalities, wherein the multiple modalities include at least two of the following: audio modality, text modality, and video modality.

[0147] Based on any of the above embodiments, the pre-trained coding model of the audio modality is obtained by training on sample speech and transcribed text;

[0148] The pre-trained encoding model of the text modality is obtained by self-supervised training of the language model based on sample text;

[0149] The pre-trained encoding model for the video modality is obtained through self-supervised training based on sample images.

[0150] Based on any of the above embodiments, the multimedia data includes audio and video data;

[0151] The defined unit includes:

[0152] Based on the feature extraction layer in the recognition model, the speech features, text features and video features of the audio and video data are extracted and fused to obtain the multimedia features of the audio and video data.

[0153] Based on the emotion recognition branch in the recognition model, emotion recognition is performed on the multimedia features to obtain the emotion recognition result of the audio and video data, and / or, based on the intent recognition branch in the recognition model, intent recognition is performed on the multimedia features to obtain the intent recognition result of the audio and video data.

[0154] Based on any of the above embodiments, the step of extracting and fusing speech features, text features, and video features of the audio and video data based on the feature extraction layer in the recognition model to obtain the multimedia features of the audio and video data includes:

[0155] The speech coding unit is used to perform speech coding on the audio of the audio and video data based on the speech coding layer in the feature extraction layer to obtain the speech features;

[0156] The speech decoding unit is used to decode the speech features based on the speech decoding layer in the feature extraction layer to obtain the transcribed text of the audio.

[0157] A semantic encoding unit is used to perform semantic encoding on the transcribed text based on the semantic encoding layer in the feature extraction layer to obtain the text features;

[0158] The video encoding unit is used to perform video encoding on the video in the audio and video data based on the video encoding layer in the feature extraction layer to obtain the video features;

[0159] The fusion coding unit is used to fuse the speech features, text features, and video features based on the fusion coding layer in the feature extraction layer to obtain the multimedia features.

[0160] Based on any of the above embodiments, the fusion coding unit includes:

[0161] The projection unit is used to project the speech features, text features and video features onto the same feature dimension based on the projection layer in the fusion coding layer, so as to obtain the projected speech features, projected text features and projected video features.

[0162] An attention unit is used to perform attention transformation on the sum of the projected speech features, projected text features, and projected video features based on the attention layer in the fusion coding layer to obtain attention features;

[0163] The residual coding unit is used to determine the multimedia features based on the residual coding layer in the fusion coding layer, by applying the sum of the projected speech features, projected text features, and projected video features, as well as the attention features.

[0164] Figure 11 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 11As shown, the electronic device may include a processor 1110, a communications interface 1120, a memory 1130, and a communication bus 1140, wherein the processor 1110, the communications interface 1120, and the memory 1130 communicate with each other via the communication bus 1140. The processor 1110 can call logical instructions in the memory 1130 to execute a recognition method, which includes: acquiring multimedia data to be recognized; determining the emotion recognition result and / or intent recognition result of the multimedia data based on a recognition model; the recognition model is obtained by performing multi-task learning, including emotion recognition and intent recognition, based on sample multimedia data and the emotion and intent labels of the sample multimedia data.

[0165] Furthermore, the logical instructions in the aforementioned memory 1130 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0166] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the recognition methods provided by the above methods. The method includes: acquiring multimedia data to be recognized; determining the emotion recognition result and / or intent recognition result of the multimedia data based on a recognition model; wherein the recognition model is obtained by performing multi-task learning, including emotion recognition and intent recognition, based on sample multimedia data and the emotion tags and intent tags of the sample multimedia data.

[0167] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program is implemented to perform the recognition methods provided by the above methods. The method includes: acquiring multimedia data to be recognized; determining the emotion recognition result and / or intent recognition result of the multimedia data based on a recognition model; wherein the recognition model is obtained by performing multi-task learning, including emotion recognition and intent recognition, based on sample multimedia data and the emotion tags and intent tags of the sample multimedia data.

[0168] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0169] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0170] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A recognition method, characterized in that, include: Acquire the multimedia data to be identified; Based on the recognition model, the emotion recognition result and / or intent recognition result of the multimedia data are determined; The recognition model is obtained by performing multi-task learning, including emotion recognition and intent recognition, based on sample multimedia data and the emotion and intent labels of the sample multimedia data. The steps for obtaining the recognition model include: Obtain an initial model, which includes an initial encoding layer, and an initial emotion recognition branch and an initial intention recognition branch respectively connected to the initial encoding layer; Based on the initial encoding layer, feature encoding is performed on the sample multimedia data to obtain the sample multimedia features of the sample multimedia data; Based on the initial emotion recognition branch, emotion recognition is performed on the multimedia features of the sample to obtain the emotion prediction result of the multimedia data of the sample. Based on the initial intent recognition branch, intent recognition is performed on the sample multimedia features to obtain the intent prediction result of the sample multimedia data; Based on the emotion prediction results, the intention prediction results, and the emotion and intention labels of the sample multimedia data, the initial model is iterated to obtain the recognition model.

2. The identification method according to claim 1, characterized in that, The steps for obtaining the initial coding layer include: The initial coding layer is constructed based on a pre-trained coding model with multiple modalities, wherein the multiple modalities include at least two of the following: audio modality, text modality, and video modality.

3. The identification method according to claim 2, characterized in that, The pre-trained encoding model for the audio modality is trained based on sample speech and transcribed text; The pre-trained encoding model of the text modality is obtained by self-supervised training of the language model based on sample text; The pre-trained encoding model for the video modality is obtained through self-supervised training based on sample images.

4. The identification method according to any one of claims 1 to 3, characterized in that, The multimedia data includes audio and video data; The determination of emotion recognition results and / or intent recognition results of the multimedia data based on the recognition model includes: Based on the feature extraction layer in the recognition model, the speech features, text features and video features of the audio and video data are extracted and fused to obtain the multimedia features of the audio and video data. Based on the emotion recognition branch in the recognition model, emotion recognition is performed on the multimedia features to obtain the emotion recognition result of the audio and video data, and / or, based on the intent recognition branch in the recognition model, intent recognition is performed on the multimedia features to obtain the intent recognition result of the audio and video data.

5. The identification method according to claim 4, characterized in that, The feature extraction layer in the recognition model extracts and fuses the speech features, text features, and video features of the audio and video data to obtain the multimedia features of the audio and video data, including: Based on the speech coding layer in the feature extraction layer, the audio of the audio and video data is speech encoded to obtain the speech features; Based on the speech decoding layer in the feature extraction layer, the speech features are decoded to obtain the transcribed text of the audio. Based on the semantic encoding layer in the feature extraction layer, the transcribed text is semantically encoded to obtain the text features; Based on the video encoding layer in the feature extraction layer, the video in the audio and video data is encoded to obtain the video features; Based on the fusion coding layer in the feature extraction layer, the speech features, text features, and video features are fused to obtain the multimedia features.

6. The identification method according to claim 5, characterized in that, The multimedia features are obtained by fusing the speech features, text features, and video features in the feature extraction layer, including: Based on the projection layer in the fusion coding layer, the speech features, text features and video features are projected onto the same feature dimension to obtain the projected speech features, projected text features and projected video features. Based on the attention layer in the fusion coding layer, attention transformation is performed on the sum of the projected speech features, projected text features, and projected video features to obtain attention features; Based on the residual coding layer in the fusion coding layer, the multimedia features are determined by applying the sum of the projected speech features, projected text features, and projected video features, as well as the attention features.

7. An identification device, characterized in that, include: The acquisition unit is used to acquire the multimedia data to be identified. The determining unit is configured to determine the emotion recognition result and / or intent recognition result of the multimedia data based on the recognition model; The recognition model is obtained by performing multi-task learning, including emotion recognition and intent recognition, based on sample multimedia data and the emotion and intent labels of the sample multimedia data. It also includes a recognition model acquisition unit, which is specifically used for: Obtain an initial model, which includes an initial encoding layer, and an initial emotion recognition branch and an initial intention recognition branch respectively connected to the initial encoding layer; Based on the initial encoding layer, feature encoding is performed on the sample multimedia data to obtain the sample multimedia features of the sample multimedia data; Based on the initial emotion recognition branch, emotion recognition is performed on the multimedia features of the sample to obtain the emotion prediction result of the multimedia data of the sample. Based on the initial intent recognition branch, intent recognition is performed on the sample multimedia features to obtain the intent prediction result of the sample multimedia data; Based on the emotion prediction results, the intention prediction results, and the emotion and intention labels of the sample multimedia data, the initial model is iterated to obtain the recognition model.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the identification method as described in any one of claims 1 to 6.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the identification method as described in any one of claims 1 to 6.

Images