Tranform architecture-based emotion category recognition method, device and equipment and medium

Abstract

The invention relates to the technical field of artificial intelligence and the field of emotion recognition, and discloses an emotion category recognition method and device based on a Transform architecture, equipment and a medium, and the method comprises the steps: obtaining a current coding feature vector through a coder; the current coding feature vector is reconstructed through a decoder, a current decoding feature vector is obtained, and the current decoding feature vector comprises a reconstruction vector corresponding to the current missing modal feature vector and a reconstruction vector corresponding to the current normal modal feature vector; determining a first weight coefficient and a second weight coefficient based on the cosine similarity; and based on the first weight coefficient and the second weight coefficient, fusing a reconstruction vector corresponding to the current missing modal feature vector and a reconstruction vector corresponding to the current normal modal feature vector to obtain a fused feature vector, and identifying the fused feature vector to obtain an emotion category of the current multi-modal data. The method is beneficial to extracting the recognition efficiency of emotion recognition.

Description

Technical Field

[0001] The present invention relates to the fields of artificial intelligence technology and emotion recognition, and in particular to an emotion category recognition method, device, equipment and medium based on a Transformer architecture. Background Art

[0002] In the field of emotion recognition, traditional research methods often rely on complete modal datasets, such as video clips that contain visual, auditory, and textual information. However, in real-world applications, current multimodal data often suffer from missing modal data due to environmental noise, sensor failures, and data transmission errors.

[0003] For example, in the current multimodal data, speech may be lost due to background noise or sensor problems, text may be unavailable due to automatic speech recognition errors or unknown vocabulary, and facial expressions may be undetectable due to lighting, motion blur, or occlusion. Since the currently missing modal data in the current multimodal data may contain key emotional information, this poses a challenge to emotion category recognition. Therefore, how to identify the emotion category of the current multimodal data in the presence of currently missing modal data is an urgent problem to be solved. Summary of the invention

[0004] The present invention provides a method, apparatus, computer device and storage medium for identifying emotion categories based on a Transformer architecture to solve the technical problem of how to identify the emotion category of current multimodal data when there is currently missing modal data in the current multimodal data.

[0005] First, a method for emotion category recognition based on a Transformer architecture is provided, including: Acquire current normal modal data and current missing modal data in the current multimodal data, and use a zero vector to fill the current missing modal data to obtain the processed current missing modal data; Performing feature extraction on the current normal modality data and the processed current missing modality data through an encoder trained in a Transformer architecture to obtain a current encoding feature vector; Reconstructing the current encoded feature vector through a decoder trained in a Transformer architecture to obtain a current decoded feature vector, wherein the current decoded feature vector includes a reconstruction vector corresponding to the current missing modality feature vector and a reconstruction vector corresponding to the current normal modality feature vector; Obtain the cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector; Determining a first weight coefficient and a second weight coefficient based on the cosine similarity and a predefined determination method; Based on the first weight coefficient and the second weight coefficient, the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector, and the fused feature vector is identified by a vector machine to obtain the emotion category of the current multimodal data.

[0006] Furthermore, the encoder trained in the Transformer architecture performs feature extraction on the current normal modality data and the processed current missing modality data to obtain a current encoding feature vector, including: Performing feature extraction on the current normal modality data through an encoder trained in a Transformer architecture to obtain a current normal modality feature vector; Feature extraction is performed on the processed current missing modality data to obtain a current missing modality feature vector, and the current normal modality feature vector and the current missing modality feature vector are concatenated to obtain a current encoding feature vector.

[0007] Furthermore, the decoder trained in the Transformer architecture reconstructs the current encoded feature vector to obtain a current decoded feature vector, wherein the current decoded feature vector includes a reconstruction vector corresponding to the current missing modality feature vector and a reconstruction vector corresponding to the current normal modality feature vector, including: Calling an input interface, and inputting the current encoded feature vector into a decoder trained in a Transformer architecture through the input interface; The current encoded feature vector is reconstructed through a decoder trained in the Transformer architecture to obtain a current decoded feature vector, wherein the current decoded feature vector includes a reconstruction vector corresponding to the current missing modality feature vector and a reconstruction vector corresponding to the current normal modality feature vector.

[0008] Furthermore, the obtaining of the cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector includes: Call the preset cosine similarity algorithm; The cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector is obtained through the cosine similarity algorithm.

[0009] Further, the determining the first weight coefficient and the second weight coefficient based on the cosine similarity and a predefined determination method includes: When the cosine similarity is less than a preset similarity, determining the reconstructed vector corresponding to the current missing modal feature vector as a key feature vector; An input mode corresponding to the key feature vector is obtained, an energy score of the input mode is obtained, and a first weight coefficient and a second weight coefficient are determined according to the energy score of the input mode.

[0010] Further, based on the first weight coefficient and the second weight coefficient, the reconstruction vector corresponding to the current missing modal feature vector and the reconstruction vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector, and the fused feature vector is identified by a vector machine to obtain the emotion category of the current multimodal data, including: Using the first weight coefficient and the second weight coefficient, a reconstruction vector corresponding to the current missing modal feature vector and a reconstruction vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector; The fused feature vector is input into a vector machine, and the fused feature vector is identified by the vector machine to obtain the emotion category of the current multimodal data.

[0011] Furthermore, before obtaining the current normal modal data and the current missing modal data in the current multimodal data and filling the current missing modal data with a zero vector to obtain the processed current missing modal data, the emotion category recognition method includes: Acquire preset normal modal data and preset missing modal data in the preset multimodal data, and use a zero vector to fill the preset missing modal data to obtain the processed preset missing modal data; By using an encoder in the Transformer architecture, feature extraction is performed on the preset normal modal data and the processed preset missing modal data to obtain a preset encoded feature vector; by using a decoder in the Transformer architecture, the preset encoded feature vector is reconstructed to obtain a preset decoded feature vector; Obtaining a first loss value between the preset encoding feature vector and the sample encoding feature vector through a first loss function, and obtaining a second loss value between the preset decoding feature vector and the preset encoding feature vector through a second loss function; The encoder and the decoder in the Transformer architecture are trained in a manner that minimizes the first loss value and the second loss value to obtain a trained encoder and a trained decoder in the Transformer architecture.

[0012] In a second aspect, a device for identifying emotion categories based on a Transformer architecture is provided, comprising: A first acquisition module is used to acquire current normal modality data and current missing modality data in the current multimodality data, and fill the current missing modality data with a zero vector to obtain the processed current missing modality data; An extraction module, configured to extract features from the current normal modality data and the processed current missing modality data through an encoder trained in a Transformer architecture to obtain a current encoded feature vector; A reconstruction module, used to reconstruct the current encoded feature vector through a decoder trained in a Transformer architecture to obtain a current decoded feature vector, wherein the current decoded feature vector includes a reconstruction vector corresponding to the current missing modality feature vector and a reconstruction vector corresponding to the current normal modality feature vector; A second acquisition module is used to obtain the cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector; A determination module, configured to determine a first weight coefficient and a second weight coefficient based on the cosine similarity and a predefined determination method; An identification module is used to fuse the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector based on the first weight coefficient and the second weight coefficient to obtain a fused feature vector, and to identify the fused feature vector through a vector machine to obtain the emotion category of the current multimodal data.

[0013] In a third aspect, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned emotion category recognition method when executing the computer program.

[0014] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned emotion category recognition method are implemented.

[0015] The present application provides a method, apparatus, computer device and storage medium for emotion category recognition based on Transformer architecture, which obtains current normal modal data and current missing modal data in current multimodal data, and uses a zero vector to fill the current missing modal data to obtain the processed current missing modal data; performs feature extraction on the current normal modal data and the processed current missing modal data through an encoder trained in the Transformer architecture to obtain a current encoded feature vector; reconstructs the current encoded feature vector through a decoder trained in the Transformer architecture to obtain a current decoded feature vector, and the current decoded feature vector includes a reconstructed vector corresponding to the current missing modal feature vector and a reconstructed vector corresponding to the current normal modal feature vector; obtains the cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector; determines a first weight coefficient and a second weight coefficient based on the cosine similarity and a predefined determination method; and based on the first weight coefficient and the second weight coefficient, converts the reconstructed vector corresponding to the current missing modal feature vector into a vector corresponding to the normal modal feature vector. The reconstructed vector corresponding to the current normal modal feature vector is fused to obtain a fused feature vector, and the fused feature vector is identified by a vector machine to obtain the emotion category of the current multimodal data. The beneficial effects are in two aspects. On the one hand, based on the first weight coefficient and the second weight coefficient, the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector, and the fused feature vector is identified by a vector machine to obtain the emotion category of the current multimodal data. In the case that the current multimodal data has the current missing modal data, the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector. Since there is an effective feature reconstruction mechanism to make up for the current missing modal data, the vector machine can more comprehensively understand the characteristics of the current missing modal data, which is conducive to improving the reliability of the identified emotion category; on the other hand, since the vector machine can automatically identify the emotion category of the current multimodal data, the recognition time of the emotion category of the current multimodal data is reduced, which is conducive to improving the recognition efficiency of the emotion category. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative labor.

[0017] Figure 1 is a schematic diagram of an application environment of a method for identifying emotion categories in one embodiment of the present invention; Figure 2 A schematic diagram of a flow chart of a method for identifying emotion categories provided by an embodiment of the present invention; Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S23; Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S25; Figure 5 yes Figure 2 A schematic flow chart of a specific implementation of step S26; Figure 6 is a structural schematic diagram of an emotion category recognition device in one embodiment of the present invention; Figure 7 It is a structural diagram of a computer device in one embodiment of the present invention. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0019] See also Figure 1 , Figure 1 is a schematic diagram of an application environment of the emotion category recognition method according to an embodiment of the present invention. The emotion category recognition method provided by the embodiment of the present invention can be applied in the following situations: Figure 1 In an application environment, a client communicates with a server through a network.

[0020] The server obtains the current normal modal data and the current missing modal data in the current multimodal data through the client, and fills the current missing modal data with a zero vector to obtain the processed current missing modal data; Performing feature extraction on the current normal modality data and the processed current missing modality data through an encoder trained in a Transformer architecture to obtain a current encoding feature vector; Reconstructing the current encoded feature vector through a decoder trained in a Transformer architecture to obtain a current decoded feature vector, wherein the current decoded feature vector includes a reconstruction vector corresponding to the current missing modality feature vector and a reconstruction vector corresponding to the current normal modality feature vector; Obtain the cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector; Determining a first weight coefficient and a second weight coefficient based on the cosine similarity and a predefined determination method; Based on the first weight coefficient and the second weight coefficient, the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector, and the fused feature vector is identified by a vector machine to obtain the emotion category of the current multimodal data.

[0021] In the scheme implemented by the above-mentioned emotion category recognition method, device, equipment and medium, the beneficial effects are in two aspects. On the one hand, based on the first weight coefficient and the second weight coefficient, the reconstruction vector corresponding to the current missing modal feature vector and the reconstruction vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector. The fused feature vector is identified by a vector machine to obtain the emotion category of the current multimodal data. In the case that the current multimodal data has current missing modal data, the reconstruction vector corresponding to the current missing modal feature vector and the reconstruction vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector. Since there is an effective feature reconstruction mechanism to make up for the current missing modal data, the vector machine can more comprehensively understand the characteristics of the current missing modal data, which is conducive to improving the reliability of the identified emotion category. On the other hand, since the vector machine can automatically identify the emotion category of the current multimodal data, the recognition time of the emotion category of the current multimodal data is reduced, which is conducive to improving the recognition efficiency of the emotion category.

[0022] The device running the client is referred to as a client device.

[0023] The device running the server is referred to as the server device.

[0024] Among them, client devices include but are not limited to smartphones, personal computers, Internet of Vehicles terminals, tablets and portable wearable devices.

[0025] The server device may be implemented by an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments. Figure 2 , Figure 2 A flowchart of a method for identifying emotion categories provided by an embodiment of the present invention includes the following steps: S21, obtaining current normal modal data and current missing modal data in the current multimodal data, and using a zero vector to fill the current missing modal data to obtain the processed current missing modal data; Among them, the current multimodal data is multimodal data that does not have complete modalities.

[0026] Exemplarily, obtaining current normal modal data and current missing modal data in current multimodal data, and using a zero vector to fill the current missing modal data to obtain the processed current missing modal data includes: Select the modal data that is not missing in the current multimodal data as the current normal modal data, and select the modal data that is missing in the current multimodal data as the current missing modal data; The current missing modal data is filled using a zero vector to obtain the processed current missing modal data.

[0027] The process of selecting the modal data that is not missing in the current multimodal data as the current normal modal data and selecting the modal data that is missing in the current multimodal data as the current missing modal data is described as follows for ease of explanation: For example, when the modal data that is not missing in the current multimodal data is the current video data and the current audio data, and the modal data that is missing in the current multimodal data is the current text data, the current video data and the current audio data are selected as the current normal modal data, and the missing current text data is selected as the current missing modal data; For example, when the modal data that is not missing in the current multimodal data is the current video data and the current text data, and the modal data that is missing in the current multimodal data is the current audio data, the current video data and the current text data are selected as the current normal modal data, and the missing current audio data is selected as the current missing modal data; For example, when the modal data that are not missing in the current multimodal data are the current text data and the current audio data, and the modal data that is missing in the current multimodal data is the current video data, the current text data and the current audio data are selected as the current normal modal data, and the missing current video data is selected as the current missing modal data.

[0028] The process of using a zero vector to fill the current missing modal data to obtain the processed current missing modal data is illustrated as follows for ease of description: For example, the current multimodal data is a data matrix, which contains the current missing modal data. In order to process the current missing modal data, the current missing modal data is filled with a zero vector, and the positions of the current missing modal data are replaced with 0. The purpose of filling is to make the dimensions of the current multimodal data complete, which is convenient for subsequent data processing and analysis.

[0029] S22, performing feature extraction on the current normal modality data and the processed current missing modality data through an encoder trained in a Transformer architecture to obtain a current encoding feature vector; Among them, the Chinese name of the Transformer architecture is: transformer architecture.

[0030] The encoder trained in the Transformer architecture performs feature extraction on the current normal modality data and the processed current missing modality data to obtain a current encoding feature vector, including: Performing feature extraction on the current normal modality data through an encoder trained in a Transformer architecture to obtain a current normal modality feature vector; Feature extraction is performed on the processed current missing modality data to obtain a current missing modality feature vector, and the current normal modality feature vector and the current missing modality feature vector are concatenated to obtain a current encoding feature vector.

[0031] The core of the Transformer architecture lies in its self-attention mechanism and multi-layer perceptron, which enable the model to capture the dependency between any two positions in the sequence without relying on sequential processing in traditional recurrent neural networks or convolutional neural networks.

[0032] S23, reconstructing the current encoding feature vector through a decoder trained in a Transformer architecture to obtain a current decoding feature vector, wherein the current decoding feature vector includes a reconstruction vector corresponding to the current missing modality feature vector and a reconstruction vector corresponding to the current normal modality feature vector; S24, obtaining the cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector; The step of obtaining the cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector includes: Call the preset cosine similarity algorithm; The cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector is obtained through the cosine similarity algorithm.

[0033] Among them, the cosine similarity algorithm is a mathematical method for calculating the similarity between the reconstructed vectors corresponding to two current missing modal feature vectors and the reconstructed vector corresponding to the current normal modal feature vector.

[0034] Among them, if the cosine similarity is larger, it means that the reconstructed vector corresponding to the current missing modal feature vector is more similar to the reconstructed vector corresponding to the current normal modal feature vector, which means that the reconstructed vector corresponding to the current missing modal feature vector does not play a key role in the emotion category.

[0035] Among them, if the cosine similarity is smaller, it means that the reconstructed vector corresponding to the current missing modal feature vector is less similar to the reconstructed vector corresponding to the current normal modal feature vector, which means that the reconstructed vector corresponding to the current missing modal feature vector plays a key role in the emotion category, and the first weight coefficient and the second weight coefficient should be adjusted.

[0036] S25, determining a first weight coefficient and a second weight coefficient based on the cosine similarity and a predefined determination method; Exemplarily, the first weight coefficient is the weight coefficient of the reconstruction vector corresponding to the current missing modal feature vector, and the second weight coefficient is the weight coefficient of the reconstruction vector corresponding to the current normal modal feature vector.

[0037] S26, based on the first weight coefficient and the second weight coefficient, the reconstruction vector corresponding to the current missing modal feature vector and the reconstruction vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector, and the fused feature vector is identified by a vector machine to obtain the emotion category of the current multimodal data.

[0038] Among them, Support Vector Machine (SVM) is a powerful supervised learning algorithm that can effectively process fused feature vectors and find the optimal classification boundary by maximizing the interval.

[0039] Wherein, before obtaining the current normal modal data and the current missing modal data in the current multimodal data, using a zero vector to fill the current missing modal data, and obtaining the processed current missing modal data, the emotion category recognition method includes: Acquire preset normal modal data and preset missing modal data in the preset multimodal data, and use a zero vector to fill the preset missing modal data to obtain the processed preset missing modal data; By using an encoder in a Transformer architecture, feature extraction is performed on the preset normal modality data and the processed preset missing modality data to obtain a preset encoding feature vector; Reconstructing the preset encoding feature vector through a decoder in the Transformer architecture to obtain a preset decoding feature vector; Obtaining a first loss value between the preset encoding feature vector and the sample encoding feature vector through a first loss function, and obtaining a second loss value between the preset decoding feature vector and the preset encoding feature vector through a second loss function; The encoder and the decoder in the Transformer architecture are trained in a manner that minimizes the first loss value and the second loss value to obtain a trained encoder and a trained decoder in the Transformer architecture.

[0040] Among them, the first loss function is: ; is the first loss value, represents the divergence loss function, represents the preset encoded feature vector output by the encoder, Represents the sample encoded feature vector output by the pre-trained network.

[0041] Among them, the first loss value is used to describe the difference between the preset coding feature vector and the sample coding feature vector. The larger the first loss value, the greater the difference between the preset coding feature vector and the sample coding feature vector. The smaller the first loss value, the smaller the difference between the preset coding feature vector and the sample coding feature vector.

[0042] The preset coding feature vector is a feature vector of the preset normal modal data and the processed preset missing modal data.

[0043] The preset decoding feature vector is a feature vector obtained by reconstructing the preset encoding feature vector.

[0044] Among them, forward means forward propagation. After the forward propagation is completed, the first loss value is obtained.

[0045] Among them, by minimizing the first loss value, it is helpful to help the encoder learn high-quality feature representations similar to those of the pre-trained network, because the pre-trained network has usually been fully trained on a large data set, so the sample encoding feature vector output by the pre-trained network often contains rich, discriminative feature information. By approximating the sample encoding feature vector through the preset encoding feature vector output by the encoder, the encoder can learn the feature extraction ability of the pre-trained network, thereby accelerating the training process of the encoder and improving the accuracy and effectiveness of the feature representation output by the encoder.

[0046] Among them, the second loss function is: ; is the second loss value, represents the divergence loss function, represents the preset decoded feature vector output by the decoder, Represents the preset encoded feature vector output by the encoder.

[0047] Among them, the second loss value is used to preset the difference between the decoding feature vector and the preset encoding feature vector. The larger the second loss value, the greater the difference between the preset decoding feature vector and the preset encoding feature vector, and the smaller the second loss value, the smaller the difference between the preset decoding feature vector and the preset encoding feature vector.

[0048] Here, backward refers to back propagation. After the back propagation is completed, the second loss value is obtained.

[0049] Among them, by minimizing the second loss value, the decoder can restore the preset coding feature vector input by the encoder as accurately as possible, which can reduce the loss of the preset coding feature vector and retain the original structure and characteristics of the preset coding feature vector.

[0050] Exemplarily, obtaining preset normal modal data and preset missing modal data in the preset multimodal data, using a zero vector to fill the preset missing modal data, and obtaining the processed preset missing modal data includes: Selecting non-missing modal data in the preset multimodal data as preset normal modal data, and selecting missing modal data in the preset multimodal data as preset missing modal data; The preset missing modal data is filled using a zero vector to obtain the processed preset missing modal data.

[0051] The process of selecting the modal data that is not missing in the preset multimodal data as the preset normal modal data and selecting the modal data that is missing in the preset multimodal data as the preset missing modal data is described as follows for ease of explanation: For example, when the modal data that is not missing in the preset multimodal data is the preset video data and the preset audio data, and the modal data that is missing in the preset multimodal data is the preset text data, the preset video data and the preset audio data are selected as the preset normal modal data, and the missing preset text data is selected as the preset missing modal data; For example, when the modal data that is not missing in the preset multimodal data is the preset video data and the preset text data, and the modal data that is missing in the preset multimodal data is the preset audio data, the preset video data and the preset text data are selected as the preset normal modal data, and the missing preset audio data is selected as the preset missing modal data; For example, when the modal data that is not missing in the preset multimodal data is the preset text data and the preset audio data, and the modal data that is missing in the preset multimodal data is the preset video data, the preset text data and the preset audio data are selected as the preset normal modal data, and the missing preset video data is selected as the preset missing modal data.

[0052] Exemplarily, the extracting features of the preset normal modality data and the processed preset missing modality data through an encoder in a Transformer architecture to obtain a preset encoded feature vector includes: Through the encoder in the preset Transformer architecture, feature extraction is performed on the preset normal modal data to obtain a preset normal modal feature vector, feature extraction is performed on the processed preset missing modal data to obtain a preset missing modal feature vector, and the preset normal modal feature vector and the preset missing modal feature vector are concatenated to obtain a preset encoding feature vector.

[0053] Exemplarily, the obtaining, by a first loss function, a first loss value between the preset encoding feature vector and the sample encoding feature vector, and the obtaining, by a second loss function, a second loss value between the preset decoding feature vector and the preset encoding feature vector, comprises: Acquire sample multimodal data corresponding to the preset multimodal data, wherein the sample multimodal data includes sample video data, sample audio data, and sample text data; Through the pre-trained network, video feature extraction is performed on the sample video data to obtain a feature vector of the sample video data, audio feature extraction is performed on the sample audio data to obtain a feature vector of the sample audio data, and text feature extraction is performed on the sample text data to obtain a feature vector of the sample text data; Concatenate the feature vector of the sample video data, the feature vector of the sample audio data, and the feature vector of the sample text data to obtain a sample coding feature vector; A first loss value between the preset encoding feature vector and the sample encoding feature vector is obtained through a first loss function, and a second loss value between the preset decoding feature vector and the preset encoding feature vector is obtained through a second loss function.

[0054] Among them, the sample multimodal data is multimodal data with complete modalities.

[0055] The sample video data is the video data in the sample multimodal data.

[0056] The sample audio data is the audio data in the sample multimodal data.

[0057] The sample text data is the text data in the sample multimodal data.

[0058] In the embodiment of the present invention, the beneficial effects lie in two aspects. On the one hand, based on the first weight coefficient and the second weight coefficient, the reconstruction vector corresponding to the current missing modal feature vector and the reconstruction vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector. The fused feature vector is identified by a vector machine to obtain the emotion category of the current multimodal data. When the current multimodal data contains current missing modal data, the reconstruction vector corresponding to the current missing modal feature vector and the reconstruction vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector. Since there is an effective feature reconstruction mechanism to make up for the current missing modal data, the vector machine can understand the characteristics of the current missing modal data more comprehensively, which is beneficial to improving the reliability of the identified emotion category. On the other hand, since the vector machine can automatically identify the emotion category of the current multimodal data, the recognition time of the emotion category of the current multimodal data is reduced, which is beneficial to improving the recognition efficiency of the emotion category.

[0059] See also Figure 3 , Figure 3 yes Figure 2 A specific implementation flow diagram of step S23 is described in detail as follows: S31, calling an input interface, and inputting the current encoding feature vector into a decoder trained in a Transformer architecture through the input interface; S32, reconstructing the current encoded feature vector through the decoder trained in the Transformer architecture to obtain a current decoded feature vector, wherein the current decoded feature vector includes a reconstruction vector corresponding to the current missing modality feature vector and a reconstruction vector corresponding to the current normal modality feature vector.

[0060] In an embodiment of the present invention, the current missing modality data is inferred through the trained decoder in the Transformer architecture to obtain a reconstruction vector corresponding to the current missing modality feature vector. The comprehensiveness of the current decoded feature vector is improved through the reconstruction vector corresponding to the current missing modality feature vector.

[0061] See also Figure 4 , Figure 4 yes Figure 2A specific implementation flow diagram of step S25 is described in detail as follows: S41, when the cosine similarity is less than a preset similarity, determining that the reconstructed vector corresponding to the current missing modal feature vector is a key feature vector; S42, obtaining an input mode corresponding to the key feature vector, obtaining an energy score of the input mode, and determining a first weight coefficient and a second weight coefficient according to the energy score of the input mode.

[0062] The step of obtaining the input mode corresponding to the key feature vector, obtaining the energy score of the input mode, and determining the first weight coefficient and the second weight coefficient according to the energy score of the input mode includes: Obtain the input mode corresponding to the key feature vector, obtain the energy score of the input mode through the energy score generation model, select the weight coefficient corresponding to the energy score of the input mode as the first weight coefficient, and select the difference obtained by subtracting the first weight coefficient from 1 as the second weight coefficient.

[0063] Among them, the energy score generation model: ; in, is the energy fraction of the mth input mode, is the mth input mode, m is the serial number of the input mode, Energy is the energy function, is the temperature parameter, log is the abbreviation of logarithm, ∑ represents the summation operation, the summation range is from k to G, k is the ordinal number of the label, G is the total number of labels, e represents the exponential function, SVM is a unimodal classifier, is the output logical value of the unimodal classifier of the mth input modality corresponding to the kth class label; in, The output logical value of the unimodal classifier representing the mth input mode corresponding to the kth class label is divided by the temperature parameter and then the result of the natural exponential function is taken.

[0064] Among them, through , which can convert the output logical value of the unimodal classifier corresponding to the k-th class label of the m-th input modality into an easy-to-handle parameter.

[0065] Among them, the energy scores of different input modes correspond to different weight coefficients.

[0066] For the sake of illustration, an example is given below: For example, if the energy score of the input mode is in the first range, coefficient 1 is used as the first weight coefficient, and if the energy score of the input mode is in the second range, coefficient 2 is used as the first weight coefficient. Coefficient 1 is greater than coefficient 2.

[0067] The input modality is evaluated by an energy score-based method to achieve the effect of dynamically adjusting the first weight coefficient. The higher the influence of the input modality on the prediction result, the higher the energy score. For multimodal data, the density functions of different modalities can be estimated by the corresponding energy functions.

[0068] The input modality refers to the type of data used when it is input into a unimodal classifier.

[0069] The input modalities include video modality, audio modality and text modality, which are three basic input forms in multimedia data processing.

[0070] The video modality refers to a data form containing a sequence of dynamic images, which are arranged in a certain time sequence and together constitute a visual scene that changes over time. The video modality contains not only static image information, but also dynamic information in the time dimension.

[0071] Among them, audio modality refers to the data form containing sound signals. Audio modality includes the frequency characteristics, amplitude characteristics, and timbre characteristics of sound. The frequency characteristics, amplitude characteristics, and timbre characteristics of sound are of great significance for the recognition, classification, and analysis of sound.

[0072] Among them, text modality refers to the data form containing characters, words, and sentences. Text modality is the basic data type in the field of natural language processing.

[0073] The process of obtaining the input mode corresponding to the key feature vector is described in detail as follows for ease of explanation: For example, when the missing modality data in the current multimodal data is the current video data, the video modality is selected as the input modality corresponding to the key feature vector.

[0074] For example, when the missing modal data in the current multimodal data is the current audio data, the audio modality is selected as the input modality corresponding to the key feature vector.

[0075] For example, when the missing modal data in the current multimodal data is the current text data, the text modality is selected as the input modality corresponding to the key feature vector.

[0076] In an embodiment of the present invention, the input modality corresponding to the key feature vector is obtained, the energy score of the input modality is obtained, and the first weight coefficient and the second weight coefficient are determined according to the energy score of the input modality. In this way, the first weight coefficient can be dynamically adjusted. This dynamic adjustment strategy enables the trained decoder to be flexibly adjusted according to the importance of the reconstructed vector corresponding to the current missing modality feature vector, thereby achieving more accurate emotion recognition.

[0077] See also Figure 5 , Figure 5 yes Figure 2 A specific implementation flow diagram of step S26 is described in detail as follows: S51, using the first weight coefficient and the second weight coefficient, to fuse the reconstruction vector corresponding to the current missing modal feature vector and the reconstruction vector corresponding to the current normal modal feature vector to obtain a fused feature vector; S52, inputting the fused feature vector into a vector machine, and using the vector machine to identify the fused feature vector to obtain the emotion category of the current multimodal data.

[0078] In an embodiment of the present invention, when there is currently missing modal data in the current multimodal data, the reconstruction vector corresponding to the currently missing modal feature vector and the reconstruction vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector. Since there is an effective feature reconstruction mechanism to make up for the currently missing modal data, the vector machine can more comprehensively understand the characteristics of the currently missing modal data, which is beneficial to improving the reliability of the identified emotion category.

[0079] See also Figure 6 , Figure 6 FIG. 1 is a schematic diagram of a structure of an emotion category recognition device according to an embodiment of the present invention. Figure 6 As shown, the emotion category recognition device includes a first acquisition module 101, an extraction module 102, a reconstruction module 103, a second acquisition module 104, a determination module 105, and an identification module 106. The detailed description of each functional module is as follows: A first acquisition module 101 is used to acquire current normal modal data and current missing modal data in current multimodal data, and fill the current missing modal data with a zero vector to obtain the processed current missing modal data; An extraction module 102 is used to extract features from the current normal modality data and the processed current missing modality data through an encoder trained in a Transformer architecture to obtain a current encoding feature vector; A reconstruction module 103 is used to reconstruct the current encoded feature vector through a decoder trained in a Transformer architecture to obtain a current decoded feature vector, wherein the current decoded feature vector includes a reconstruction vector corresponding to the current missing modality feature vector and a reconstruction vector corresponding to the current normal modality feature vector; The second acquisition module 104 is used to obtain the cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector; A determination module 105, configured to determine a first weight coefficient and a second weight coefficient based on the cosine similarity and a predefined determination method; The identification module 106 is used to fuse the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector based on the first weight coefficient and the second weight coefficient to obtain a fused feature vector, and identify the fused feature vector through a vector machine to obtain the emotion category of the current multimodal data.

[0080] In the embodiment of the present invention, the beneficial effects lie in two aspects. On the one hand, based on the first weight coefficient and the second weight coefficient, the reconstruction vector corresponding to the current missing modal feature vector and the reconstruction vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector. The fused feature vector is identified by a vector machine to obtain the emotion category of the current multimodal data. When the current multimodal data contains current missing modal data, the reconstruction vector corresponding to the current missing modal feature vector and the reconstruction vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector. Since there is an effective feature reconstruction mechanism to make up for the current missing modal data, the vector machine can understand the characteristics of the current missing modal data more comprehensively, which is beneficial to improving the reliability of the identified emotion category. On the other hand, since the vector machine can automatically identify the emotion category of the current multimodal data, the recognition time of the emotion category of the current multimodal data is reduced, which is beneficial to improving the recognition efficiency of the emotion category.

[0081] For the specific limitations of the emotion category recognition device, please refer to the limitations of the emotion category recognition method above, which will not be repeated here.

[0082] Each module in the above-mentioned emotion category recognition device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.

[0083] See also Figure 7 , Figure 7 1 is another structural diagram of a computer device in one embodiment of the present invention. In one embodiment, a computer device is provided. The computer device is a server device or a client device. The internal structure diagram thereof can be as shown in FIG. Figure 7As shown. The computer device includes a processor, a memory, a network interface, a display screen and an input device connected via a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external database. When the computer program is executed by the processor, the functions or steps of an emotion category recognition method based on the Transformer architecture can be implemented.

[0084] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor.

[0085] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can refer to the relevant description of the aforementioned method embodiment. In order to avoid repetition, they will not be described one by one here.

[0086] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), and a network processor (NP); it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components.

[0087] The above description and the accompanying drawings sufficiently illustrate the embodiments of the present disclosure to enable those skilled in the art to practice them. In this article, each embodiment may focus on the differences from other embodiments, and the same or similar parts between the embodiments may refer to each other. For the methods and products disclosed in the embodiments, if they correspond to the method part disclosed in the embodiments, then the relevant parts can refer to the description of the method part.

[0088] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software may depend on the specific application and design constraints of the technical solution. Technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the embodiments of the present disclosure. Technicians can clearly understand that for the convenience and simplicity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0089] In the embodiments disclosed herein, the disclosed methods and products (including but not limited to devices and equipment) can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of units can be only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some sub-samples can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between each other shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms. The units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the units may be selected according to actual needs to implement this embodiment. In addition, each functional unit in the embodiment of the present disclosure may be integrated in a processing unit, or each unit may exist physically alone, or two or more units may be integrated in one unit.

[0090] The flowchart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to the embodiment of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of the code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. In some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. In the description corresponding to the flowchart and the block diagram in the accompanying drawings, the operations or steps corresponding to different boxes can also occur in a different order from the order disclosed in the description, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, which can depend on the functions involved. Each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by a dedicated hardware-based system that performs the specified functions or actions, or may be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for emotion category recognition based on Transformer architecture, characterized in that: include: Acquire current normal modal data and current missing modal data in the current multimodal data, and use a zero vector to fill the current missing modal data to obtain the processed current missing modal data; Performing feature extraction on the current normal modality data and the processed current missing modality data through an encoder trained in a Transformer architecture to obtain a current encoding feature vector; Reconstructing the current encoded feature vector through a decoder trained in a Transformer architecture to obtain a current decoded feature vector, wherein the current decoded feature vector includes a reconstruction vector corresponding to the current missing modality feature vector and a reconstruction vector corresponding to the current normal modality feature vector; Obtain the cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector; Determining a first weight coefficient and a second weight coefficient based on the cosine similarity and a predefined determination method; Based on the first weight coefficient and the second weight coefficient, the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector, and the fused feature vector is identified by a vector machine to obtain the emotion category of the current multimodal data.

2. The emotion category recognition method according to claim 1, characterized in that: The encoder trained in the Transformer architecture performs feature extraction on the current normal modality data and the processed current missing modality data to obtain a current encoding feature vector, including: Performing feature extraction on the current normal modality data through an encoder trained in a Transformer architecture to obtain a current normal modality feature vector; Feature extraction is performed on the processed current missing modality data to obtain a current missing modality feature vector, and the current normal modality feature vector and the current missing modality feature vector are concatenated to obtain a current encoding feature vector.

3. The emotion category recognition method according to claim 1, characterized in that: The decoder trained in the Transformer architecture reconstructs the current encoded feature vector to obtain a current decoded feature vector, wherein the current decoded feature vector includes a reconstruction vector corresponding to the current missing modality feature vector and a reconstruction vector corresponding to the current normal modality feature vector, including: Calling an input interface, and inputting the current encoded feature vector into a decoder trained in a Transformer architecture through the input interface; The current encoded feature vector is reconstructed through a decoder trained in the Transformer architecture to obtain a current decoded feature vector, wherein the current decoded feature vector includes a reconstruction vector corresponding to the current missing modality feature vector and a reconstruction vector corresponding to the current normal modality feature vector.

4. The emotion category recognition method according to claim 1, characterized in that: The obtaining of the cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector includes: Call the preset cosine similarity algorithm; The cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector is obtained through the cosine similarity algorithm.

5. The emotion category recognition method according to claim 1, characterized in that: The determining of the first weight coefficient and the second weight coefficient based on the cosine similarity and a predefined determination method includes: When the cosine similarity is less than a preset similarity, determining the reconstructed vector corresponding to the current missing modal feature vector as a key feature vector; An input mode corresponding to the key feature vector is obtained, an energy score of the input mode is obtained, and a first weight coefficient and a second weight coefficient are determined according to the energy score of the input mode.

6. The emotion category recognition method according to claim 1, characterized in that: The method of fusing the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector based on the first weight coefficient and the second weight coefficient to obtain a fused feature vector, and identifying the fused feature vector through a vector machine to obtain the emotion category of the current multimodal data includes: Using the first weight coefficient and the second weight coefficient, a reconstruction vector corresponding to the current missing modal feature vector and a reconstruction vector corresponding to the current normal modal feature vector are fused to obtain a fused feature vector; The fused feature vector is input into a vector machine, and the fused feature vector is identified by the vector machine to obtain the emotion category of the current multimodal data.

7. The emotion category recognition method according to claim 1, characterized in that: Before obtaining the current normal modal data and the current missing modal data in the current multimodal data and filling the current missing modal data with a zero vector to obtain the processed current missing modal data, the emotion category recognition method includes: Acquire preset normal modal data and preset missing modal data in the preset multimodal data, and use a zero vector to fill the preset missing modal data to obtain the processed preset missing modal data; By using an encoder in a Transformer architecture, feature extraction is performed on the preset normal modality data and the processed preset missing modality data to obtain a preset encoding feature vector; Reconstructing the preset encoding feature vector through a decoder in the Transformer architecture to obtain a preset decoding feature vector; Obtaining a first loss value between the preset encoding feature vector and the sample encoding feature vector through a first loss function, and obtaining a second loss value between the preset decoding feature vector and the preset encoding feature vector through a second loss function; The encoder and the decoder in the Transformer architecture are trained in a manner that minimizes the first loss value and the second loss value to obtain a trained encoder and a trained decoder in the Transformer architecture.

8. An emotion category recognition device based on Transformer architecture, characterized in that: include: A first acquisition module is used to acquire current normal modal data and current missing modal data in the current multimodal data, and use a zero vector to fill the current missing modal data to obtain the processed current missing modal data; An extraction module, configured to extract features from the current normal modality data and the processed current missing modality data through an encoder trained in a Transformer architecture to obtain a current encoding feature vector; A reconstruction module, used to reconstruct the current encoded feature vector through a decoder trained in a Transformer architecture to obtain a current decoded feature vector, wherein the current decoded feature vector includes a reconstruction vector corresponding to the current missing modality feature vector and a reconstruction vector corresponding to the current normal modality feature vector; A second acquisition module is used to obtain the cosine similarity between the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector; A determination module, configured to determine a first weight coefficient and a second weight coefficient based on the cosine similarity and a predefined determination method; An identification module is used to fuse the reconstructed vector corresponding to the current missing modal feature vector and the reconstructed vector corresponding to the current normal modal feature vector based on the first weight coefficient and the second weight coefficient to obtain a fused feature vector, and to identify the fused feature vector through a vector machine to obtain the emotion category of the current multimodal data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the emotion category recognition method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the emotion category recognition method according to any one of claims 1 to 7 are implemented.

Images