Emotion classification model training method, emotion classification method, device, electronic equipment, computer readable storage medium and computer program product

Abstract

The application provides a mood classification model training method and device, electronic equipment, computer readable storage medium and computer program product. The method comprises: adding an audio prompt vector and a text prompt vector in a conversion layer of a mood classification model; converting an audio feature vector of a sample object through the conversion layer to obtain a converted audio feature vector, and converting a text feature vector of the sample object through the conversion layer to obtain a converted text feature vector; classifying through a linear layer of the mood classification model based on the converted audio feature vector and the converted text feature vector to obtain a mood classification result of the sample object; and updating the audio prompt vector and the text prompt vector and the linear layer based on an error between the mood classification result and a pre-labeled label. Through the application, the accuracy of the trained mood classification model can be improved while reducing the model training cost.

Description

Technical Field

[0001] This application relates to the field of information processing, and in particular to training methods for emotion classification models, emotion classification methods, devices, electronic devices, computer-readable storage media, and computer program products. Background Technology

[0002] Emotion recognition is widely used in intelligent telemarketing and customer service scenarios. Some emotion classification models classify emotions based on text content, but due to the lack of audio information, the accuracy of the model in classifying emotions is low. Other emotion classification models use fine-tuning training methods to perform emotion recognition using both text and audio information. However, this model training method requires a large amount of data and is costly. Summary of the Invention

[0003] This application provides a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for an emotion classification model, which can improve the accuracy of the trained emotion classification model while reducing the model training cost.

[0004] The technical solution of this application embodiment is implemented as follows:

[0005] This application provides a method for training an emotion classification model, including:

[0006] Add audio cue vectors and text cue vectors to the transformation layer of the emotion classification model;

[0007] The audio feature vector of the sample object is transformed by the transformation layer to obtain the transformed audio feature vector, and the text feature vector of the sample object is transformed by the transformation layer to obtain the transformed text feature vector.

[0008] Based on the converted audio feature vector and the converted text feature vector, the emotion classification model is used to classify the sample object through a linear layer to obtain the emotion classification result.

[0009] Based on the error between the emotion classification result and the pre-labeled tags, the audio cue vector, the text cue vector, and the linear layer are updated.

[0010] This application also provides an emotion classification method, including:

[0011] For an object whose emotion needs to be classified, an emotion classification model is invoked based on the object's text feature vector and audio feature vector to perform classification and obtain the emotion classification result of the object.

[0012] This application provides a training device for an emotion classification model, comprising:

[0013] Add a module to add audio cue vectors and text cue vectors to the transformation layer of the emotion classification model;

[0014] The conversion module is used to convert the audio feature vector of the sample object through the conversion layer to obtain the converted audio feature vector, and to convert the text feature vector of the sample object through the conversion layer to obtain the converted text feature vector;

[0015] The prediction module is used to classify the sample object based on the converted audio feature vector and the converted text feature vector through the linear layer included in the emotion classification model, so as to obtain the emotion classification result of the sample object.

[0016] An update module is used to update the audio cue vector, the text cue vector, and the linear layer based on the error between the emotion classification result and the pre-labeled label.

[0017] This application embodiment also provides an emotion classification device, including:

[0018] The classification module is used to classify objects whose emotions are to be classified, based on the text feature vector and audio feature vector of the object, by calling the emotion classification model to obtain the emotion classification result of the object.

[0019] This application provides an electronic device, including:

[0020] Memory is used to store executable instructions or computer programs.

[0021] When the processor executes computer-executable instructions or computer programs stored in the memory, it implements the training method or emotion classification method of the emotion classification model provided in the embodiments of this application.

[0022] This application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which, when executed by a processor, implements the training method or emotion classification method of the emotion classification model provided in this application.

[0023] This application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, they implement the training method or emotion classification method of the emotion classification model provided in this application.

[0024] The embodiments of this application have the following beneficial effects:

[0025] By fixing the parameters of the pre-trained converter model included in the emotion classification model, and updating only the linear layer, audio cue vector, and text cue vector included in the emotion classification model based on the error between the predicted emotion classification result and the pre-labeled label, the number of parameters that need to be adjusted during model training is reduced. In this way, the amount of data required for model training can be effectively reduced, thereby reducing the training cost of the model. In addition, since the model training process involves information from both audio and text dimensions, classifying information from both dimensions can improve the accuracy of the trained emotion classification model. Attached Figure Description

[0026] Figure 1 This is a schematic diagram of the architecture of the training system 100 for the emotion classification model provided in this application embodiment;

[0027] Figure 2A This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application;

[0028] Figure 2B This is a schematic diagram of the structure of the electronic device 600 provided in the embodiments of this application;

[0029] Figure 3 This is a flowchart illustrating the training method of the emotion classification model provided in the embodiments of this application;

[0030] Figure 4 This is a flowchart illustrating the training method of the emotion classification model provided in the embodiments of this application;

[0031] Figure 5 This is a flowchart illustrating the training method of the emotion classification model provided in the embodiments of this application;

[0032] Figure 6 This is a schematic diagram illustrating the principle of the training method for the emotion classification model provided in the embodiments of this application. Detailed Implementation

[0033] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0034] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0035] It is understood that in the embodiments of this application, data related to user information (such as audio features, text features, etc.) are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with relevant laws, regulations and standards.

[0036] In the embodiments of this application, the terms "layer" or "unit" refer to a computer program or part of a computer program with a predetermined function, which works together with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more layers or units. Furthermore, each layer or unit can be part of an overall layer or unit that includes the functionality of that layer or unit.

[0037] In the following description, the terms “first, second, ...” are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that “first, second, ...” may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0039] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0040] 1) Prompt tuning: Prompt tuning uses trainable prompts to guide the model to output the results required for the task. These prompts can be text, audio, or vector information, used to help the model understand a specific task or requirement.

[0041] 2) Fine-tuning: Fine-tuning is mainly used to fine-tune a pre-trained model for a specific task, requiring training all parameters of the model. When it comes to a specific downstream task (e.g., sentiment analysis, text classification, etc.), due to the generalization ability of the pre-trained model, it is not necessary to train from scratch.

[0042] 3) Transformer model: The Transformer model is a deep learning model based on the self-attention mechanism. It uses the self-attention mechanism to capture long-distance dependencies in sequence data and uses an encoder-decoder structure to handle the sequence-to-sequence transformation problem.

[0043] 4) Self-attention Mechanism: Self-attention is a widely used mechanism in sequence processing, especially in Natural Language Processing (NLP). Given a sequence (such as a sequence of words in a sentence), each element (such as a word) references all other elements in the sequence when representing itself. The core of self-attention lies in calculating a weight matrix, which represents the degree of "attention" each element in the sequence gives to other elements. This weight matrix is ​​calculated by the dot product of three vectors: query, key, and value. Self-attention can capture long-distance dependencies in a sequence and establish associations between different positions.

[0044] Emotion classification is widely used in intelligent telemarketing and customer service scenarios. Based on the text and voice information in the customer's conversation, the customer's emotions can be classified to understand whether the customer is currently experiencing negative emotions and whether reassurance or other actions are needed.

[0045] In related technologies, emotion classification models primarily rely on text content for classification. However, these methods suffer from low accuracy due to the lack of audio information. For instance, in some situations, a customer may be extremely angry, but this cannot be adequately reflected in the text alone. The same text, "I don't need it," conveys completely different emotions depending on whether the customer says it angrily or politely. Therefore, incorporating audio information into emotion classification models is crucial for improving accuracy.

[0046] In addition, the emotion classification models provided by related technologies also use text and sound information at the same time, and use fine-tuning to fit the pre-trained text and sound to the emotion classification task. However, this training method requires a large amount of training data and the training level is inconsistent between different modalities.

[0047] Based on this, embodiments of this application provide a training method, apparatus, electronic device, computer-readable storage medium, and computer program product for an emotion classification model, which can improve the accuracy of the trained emotion classification model while reducing the model training cost. The electronic device provided in this application can be implemented as a server, or implemented collaboratively by a server and a terminal. The following description uses an example of a training method for the emotion classification model provided in this application, implemented collaboratively by a server and a terminal.

[0048] For example, see Figure 1 , Figure 1 This is a schematic diagram of the architecture of the training system 100 for the emotion classification model provided in this application embodiment. To support the training and application of an emotion classification model, such as... Figure 1 As shown, the training system 100 for the emotion classification model includes: a server 200, a network 300, and a terminal 400. The terminal 400 is connected to the server 200 through the network 300. The network 300 can be a local area network (LAN), a wide area network (WAN), or a combination of both. The terminal 400 is a user-associated terminal, on which a client 410 runs. The client 410 can be of various types, such as a dedicated model training client, a smart telemarketing and customer service related client, or a browser.

[0049] In some embodiments, server 200 can extract features from sample objects to obtain audio and text features of the sample objects. Next, server 200 performs embedding processing on the audio and text features of the sample objects to obtain audio feature vectors and text feature vectors. Then, server 200 adds audio and text cue vectors to the transformation layer of the emotion classification model, and transforms the audio feature vector of the sample objects through the transformation layer to obtain transformed audio feature vectors, and transforms the text feature vector of the sample objects through the transformation layer to obtain transformed text feature vectors. Subsequently, server 200 classifies the sample objects based on the transformed audio and text feature vectors through the linear layer of the emotion classification model to obtain the emotion classification result of the sample objects. Next, server 200 updates the audio and text cue vectors, as well as the linear layer, based on the error between the emotion classification result and the pre-labeled labels, thereby obtaining the trained emotion classification model. Finally, server 200 can send the trained emotion classification model to terminal 400 via network 300, i.e., the trained emotion classification model can be deployed on client 410 of terminal 400.

[0050] It should be noted that the technical solutions provided in this application can be applied to various application scenarios, including intelligent telemarketing, intelligent customer service, social media detection, public opinion analysis, and smart home scenarios. For example, a trained emotion classification model can be deployed in an intelligent customer service client to classify emotions based on the customer's audio and text features, accurately determining the customer's current emotion and providing corresponding intelligent services. Alternatively, the trained emotion classification model can be applied to social media detection scenarios, where brand and market analysts use the model to detect public sentiment on social media, thereby understanding public opinions on their products or services, and helping companies promptly identify potential problems and take necessary measures.

[0051] In other embodiments, the embodiments of this application can also be implemented with the aid of cloud technology, which refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize the computation, storage, processing, and sharing of data.

[0052] Cloud technology is a general term encompassing network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form resource pools, allowing for on-demand use with flexibility and convenience. Cloud computing technology will become a crucial support. The backend services of cloud computing systems require substantial computing and storage resources.

[0053] Example, Figure 1 The server 200 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. The terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle terminal, etc., but is not limited to these. The terminal 400 and server 200 can be directly or indirectly connected via wired or wireless communication, which is not limited in this embodiment.

[0054] The structure of the electronic device used for implementing the training method of the emotion classification model provided in the embodiments of this application will be described below. Taking the electronic device 500 as a server as an example, see... Figure 2A , Figure 2A This is a schematic diagram of the structure of the electronic device 500 provided in the embodiments of this application. Figure 2AThe illustrated electronic device 500 includes at least one processor 510, a memory 540, and at least one network interface 520. The various components in the electronic device 500 are coupled together via a bus system 530. It is understood that the bus system 530 is used to implement communication between these components. In addition to a data bus, the bus system 530 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2A The general labeled all buses as Bus System 530.

[0055] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0056] The memory 540 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 540 may optionally include one or more storage devices physically located away from the processor 510.

[0057] The memory 540 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 540 described in this application embodiment is intended to include any suitable type of memory.

[0058] In some embodiments, memory 540 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0059] Operating system 541 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, for implementing various basic business functions and handling hardware-based tasks;

[0060] The network communication module 542 is used to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc.

[0061] In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2A A training device 543 for an emotion classification model stored in memory 540 is shown. It can be software in the form of programs and plug-ins, including the following software modules: addition module 5431, transformation module 5432, prediction module 5433, and update module 5434. These modules are logical and can therefore be arbitrarily combined or further split according to the functions they implement.

[0062] The structure of the electronic device for implementing the emotion classification method provided in the embodiments of this application will be further described below, taking the electronic device 600 as a server as an example. See below. Figure 2B , Figure 2B This is a schematic diagram of the structure of the electronic device 600 provided in the embodiments of this application. Figure 2B The electronic device 600 shown includes an emotion classification device 643, which can be software in the form of programs and plug-ins, including the following software modules: classification module 6431, the functions of each module will be described below.

[0063] It should be noted that, Figure 2B The illustrated electronic device 600 includes a processor 610, a network interface 620, a bus system 630, a memory 640, an operating system 641, and a network communication module 642, all of which are related to... Figure 2A The corresponding modules contained therein have the same structure and the same function, and will not be described again in the embodiments of this application.

[0064] The training method of the emotion classification model provided in this application embodiment will be specifically described below with reference to the exemplary application and implementation of the server provided in the embodiments of this application.

[0065] The training method of the emotion classification model provided in this application embodiment will be specifically described below with reference to the exemplary application and implementation of the terminal device provided in the embodiments of this application.

[0066] See Figure 3 , Figure 3 This is a flowchart illustrating the training method of the emotion classification model provided in this application embodiment, which will be combined with... Figure 3 The steps shown are explained.

[0067] In step 101, audio cue vectors and text cue vectors are added to the transformation layer of the emotion classification model.

[0068] Here, audio and text cue vectors to be trained can be added to the transformation layer of the emotion classification model. The emotion classification model can be trained using a cue-optimized training method. The output of the model can be guided by updating the audio and text cue vectors. This does not require updating a large number of model parameters, thus effectively reducing the training cost of the model.

[0069] In step 102, the audio feature vector of the sample object is transformed by the transformation layer to obtain the transformed audio feature vector, and the text feature vector of the sample object is transformed by the transformation layer to obtain the transformed text feature vector.

[0070] It should be noted that the series of transformation operations performed in the transformation layer can be linear transformations or non-linear transformations (such as convolutional layers, recurrent layers, or attention mechanism layers), without any specific limitations here.

[0071] For example, assuming a convolutional layer is used, it can be used to extract time-frequency features when processing audio feature vectors, and to extract relationships between words when processing text feature vectors. Assuming a self-attention mechanism layer is used, the multi-head attention layer in the Transformer can simultaneously focus on different parts of the sequence, extracting dependencies between features. Assuming a fully connected layer is used, the input features can be transformed into output features of arbitrary dimensions. Thus, through the above transformation process, the original audio and text feature vectors are transformed into a form more suitable for machine learning models, thereby effectively improving the model's performance and generalization ability.

[0072] In some embodiments, the conversion layer described above may include a first attention mechanism and a second attention mechanism, see [link to relevant documentation]. Figure 4 , Figure 4 This is a flowchart illustrating the training method of the emotion classification model provided in this application embodiment, as shown below. Figure 4 As shown, Figure 3 Step 102 shown can be achieved through Figure 4 Steps 1021 to 1022 shown are implemented, and will be combined with Figure 4 The steps shown are explained.

[0073] In step 1021, the audio feature vector is transformed through the first attention mechanism to obtain the transformed audio feature vector.

[0074] It's important to note that the attention mechanism here is a mechanism that allows the model to dynamically allocate attention based on the importance of different parts of the input data. This attention mechanism can take various forms, such as multi-head self-attention or sequence-to-sequence attention. Multi-head self-attention is a key component of the Transformer model; it divides the input sequence into multiple parts, calculates attention independently for each part, and then combines all the attention outputs to obtain the final sequence representation. Sequence attention is typically used in encoder-decoder architectures, enabling the encoder to focus on the sequence generated by the decoder to more accurately generate the target sequence.

[0075] In some embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating the training method of the emotion classification model provided in this application embodiment, as shown below. Figure 5 As shown, Figure 4 Step 1021 shown can be achieved through Figure 5 Steps 10211 to 10213 shown are implemented by combining Figure 5 The steps shown are explained.

[0076] In step 10211, the audio cue vector and the audio feature vector are concatenated to obtain the first concatenated vector.

[0077] For example, suppose the audio cue vector might contain information such as style (a string, such as "melodious," "noisy," etc.), emotion (a word describing emotion, such as "happy," "sad," etc.), and timbre (a word indicating a specific type of sound, such as "piano," "noise," etc.). Here, the audio cue vector can be a discrete label or a continuous numerical value; no specific limitation is made here. Suppose the audio feature vector might contain information such as pitch (i.e., the volume of the audio) and tempo (i.e., the speed of playback or speaking in the audio). Next, the audio cue vector is concatenated to the audio feature vector. The specific concatenation scheme can be to adjust the dimension of the audio cue vector to match the dimension of the audio feature vector, and then concatenate the audio cue vector and the audio feature vector in sequence. Alternatively, the audio cue vector can be transformed in some form before concatenation to better integrate with the audio feature vector. The specific concatenation method can be determined according to the needs of the model; no specific limitation is made here. Finally, the first concatenated vector is obtained, which is a composite vector containing information from the audio cue vector and the audio feature vector.

[0078] In step 10212, based on the first concatenation vector and the audio feature vector, the first query vector, the first key vector, and the first value vector of the first attention mechanism are determined.

[0079] It should be noted that in the first attention mechanism, the first query vector describes the information that the model wants to know, or the part that the model wants to focus on and find when exploring the input data; the first key vector contains the information that the model already knows, or the part that the model uses to index information in the input data; the first value vector represents the information associated with the key vector, or the "value" that the first key vector points to. That is, the first value vector contains the actual information that the model needs to utilize, such as words in text or frequencies in audio.

[0080] In some embodiments, step 10212 described above can be implemented as follows: multiplying the audio feature vector with the first weight matrix and using the first multiplication result as the first query vector; multiplying the first concatenated vector with the second weight matrix and using the second multiplication result as the first key vector; multiplying the first concatenated vector with the third weight matrix and using the third multiplication result as the first value vector.

[0081] It should be noted that the audio conversion layer consists of audio cue vectors and a pre-trained converter model. The first weight matrix, the second weight matrix, and the third weight matrix are determined by the pre-trained converter model during the pre-training process. The first weight matrix is ​​the weight matrix corresponding to the first query vector, the second weight matrix is ​​the weight matrix corresponding to the first key vector, and the third weight matrix is ​​the weight matrix corresponding to the first value vector.

[0082] For example, suppose the audio cue vector is P. v The audio feature vector is X v The first weight matrix obtained in the pre-trained converter model is W. Qv The second weight matrix is ​​W Kv The third weight matrix is ​​W Vv First, the audio cue vector P v With audio feature vector X v Perform concatenation, assuming the first concatenated vector is P_X. v , the audio feature vector X v With the first weight matrix W Qv Perform the multiplication to obtain the first multiplication result X. v ·W Qv That is, the first query vector Q v For X v ·W Qv ; the first concatenated vector P_X v With the second weight matrix W KvPerform the multiplication to obtain the second multiplication result P_X v ·W Kv That is, the first key vector K v For P_X v ·W Kv ; the first concatenated vector P_X v With the third weight matrix W Vv Perform the multiplication to obtain the third multiplication result P_X v ·W Vv That is, the first value vector V v For P_X v ·W Vv .

[0083] In step 10213, the converted audio feature vector is determined based on the first query vector, the first key vector, and the first value vector.

[0084] In some embodiments, step 10213 described above can be implemented as follows: transpose the first key vector to obtain a transposed first key vector; multiply the transposed first key vector with the first query vector to obtain a fourth multiplication result; divide the fourth multiplication result by the square root of the dimension of the first key vector to obtain a first division result; normalize the first division result to obtain a normalized first division result; multiply the normalized first division result with the first value vector, and use the resulting fifth multiplication result as the converted audio feature vector.

[0085] For example, suppose the first query vector is Q. v The first key vector is K v The first value vector is V v The dimension of the first key vector is dk. v For the first key vector K v Transpose the vector to obtain the first key vector K. v T For the transposed first key vector K v T With the first query vector Q v Perform the multiplication to obtain the fourth multiplication result Q. v K v T The result of the fourth multiplication is Q. v K v T The square root of the dimension of the first key vector Perform the division to obtain the first division result. Use the softmax function to process the first division result Normalization is performed to obtain the normalized first division result. Finally, the normalized first phase division result With the first value vector V v Perform the multiplication to obtain the fifth multiplication result: softmax()W v That is, the converted audio feature vector is

[0086] In step 1022, the text feature vector is transformed through the second attention mechanism to obtain the transformed text feature vector.

[0087] Here, the first attention mechanism and the second attention mechanism can be the same or different, and no specific restrictions are made here.

[0088] It should be noted that the implementation of step 1022 in this embodiment is similar to the implementation of step 1021 described above. For details, please refer to the implementation of step 1021 described above, which will not be repeated here.

[0089] In some embodiments, step 1022 described above can be implemented as follows: concatenating the text prompt vector and the text feature vector to obtain a second concatenated vector; determining the second query vector, the second key vector, and the second value vector of the second attention mechanism based on the second concatenated vector and the text feature vector; and determining the transformed text feature vector based on the second query vector, the second key vector, and the second value vector.

[0090] In some embodiments, the above-described determination of the second query vector, second key vector, and second value vector of the second attention mechanism based on the second concatenation vector and the text feature vector can be achieved in the following manner: multiplying the text feature vector with the fourth weight matrix and using the sixth multiplication result as the second query vector; multiplying the second concatenation vector with the fifth weight matrix and using the seventh multiplication result as the second key vector; multiplying the second concatenation vector with the sixth weight matrix and using the eighth multiplication result as the second value vector.

[0091] It should be noted that the fourth weight matrix is ​​the weight matrix corresponding to the second query vector, the fifth weight matrix is ​​the weight matrix corresponding to the second key vector, and the sixth weight matrix is ​​the weight matrix corresponding to the second value vector.

[0092] For example, suppose the text prompt vector is P. t The text feature vector is X t The fourth weight matrix obtained in the pre-trained converter model is W. Qt The fifth weight matrix is ​​W Kt The sixth weight matrix is ​​W Vt First, the text prompt vector P t With text feature vector Xt Perform concatenation, assuming the resulting second concatenated vector is P_X. t The text feature vector X t With the fourth weight matrix W Qt Performing the multiplication, we obtain the sixth multiplication result X. t ·W Qt That is, the second query vector Q t For X t ·W Qt ; the second concatenation vector P_X t With the fifth weight matrix W Kt Performing the multiplication, we obtain the seventh multiplication result P_X. t ·W Kt That is, the second key vector K t For P_X t ·W Kt ; the second concatenation vector P_X t With the sixth weight matrix W Vt Performing the multiplication, we obtain the eighth multiplication result P_X. t ·W Vt That is, the second value vector V t For P_X t ·W Vt .

[0093] In some embodiments, the above-mentioned determination of the transformed text feature vector based on the second query vector, the second key vector, and the second value vector can be achieved in the following manner: transpose the second key vector to obtain the transposed second key vector; multiply the transposed second key vector with the second query vector to obtain the ninth multiplication result; divide the ninth multiplication result by the square root of the dimension of the second key vector to obtain the second division result; normalize the second division result to obtain the normalized second division result; multiply the normalized second division result with the second value vector, and use the resulting tenth multiplication result as the transformed text feature vector.

[0094] For example, suppose the second query vector is Q. t The second key vector is K. t The second value vector is V t The dimension of the second key vector is dk. t For the second key vector K t Transpose the vector to obtain the second key vector K. t T For the transposed second key vector K t T With the second query vector Q t Multiplying them together, we get the ninth product Q. t K tT Multiply the ninth result Q t K t T The square root of the second key vector dimension Perform a division to obtain the second division result. Use the softmax function to analyze the second division result. Normalization is performed to obtain the normalized second division result. Finally, the normalized first phase division result With the second value vector V t Multiplying them together, we get the tenth product. That is, the converted text feature vector is

[0095] In some embodiments, before performing step 102, the following processes may also be performed: feature extraction of the sample object to obtain audio features and text features of the sample object; embedding of the audio features and text features of the sample object to obtain audio feature vectors and text feature vectors.

[0096] For example, suppose there is a task to extract features from user comments on social media, including both voice and text comments. First, the voice comments are converted into audio signals. Then, audio features are extracted using audio processing techniques (such as Short-Time Fourier Transform, Mel-frequency cepstral coefficients, etc.). These audio features may include pitch, volume, frequency distribution, etc. Simultaneously, for the text comments, text features can be extracted using techniques such as the bag-of-words model, TF-IDF, or word embeddings (such as Word2Vec, GloVe). These text features contain the semantic content of the text. Next, the extracted audio features are embedded (e.g., through embedding layers) to obtain the audio feature vector of the user's audio comment, and the extracted text features are embedded (e.g., using word embedding techniques) to obtain the text feature vector of the user's text comment. In this way, the raw audio and text data are transformed into a structured feature representation that can be processed by machine learning models, helping sentiment classification models to accurately predict user emotions.

[0097] In step 103, based on the transformed audio feature vector and the transformed text feature vector, the emotion classification is performed through the linear layer of the emotion classification model to obtain the emotion classification result of the sample object.

[0098] In some embodiments, the above-mentioned classification of the sample object based on the transformed audio feature vector and the transformed text feature vector through the linear layer of the emotion classification model to obtain the emotion classification result of the sample object can be achieved in the following way: concatenating the transformed audio feature vector and the transformed text feature vector to obtain a third concatenated vector; multiplying the third concatenated vector with the first parameter in the linear layer to obtain an eleventh multiplication result; adding the eleventh multiplication result with the second parameter in the linear layer to obtain an addition result; normalizing the addition result, and using the normalized addition result as the emotion classification result of the sample object.

[0099] For example, assume the converted audio feature vector is R. v The transformed text feature vector is R. t The converted audio feature vector R v The transformed text feature vector is R t The concatenation process yields the third concatenated vector X. Assuming the first parameter of the linear layer is w and the second parameter is b, the third concatenated vector X is multiplied by the first parameter w in the linear layer, resulting in the eleventh multiplication result wX. The first multiplication result is then added to the second parameter b in the linear layer, resulting in the summation result wX+b. The summation result is then normalized using the softmax function, resulting in the normalized summation result softmax(wX+b). This is the sentiment classification result of the sample object, softmax(wX+b).

[0100] In step 104, the audio cue vector, text cue vector, and linear layer are updated based on the error between the emotion classification result and the pre-labeled labels.

[0101] Here, the pre-trained transformer model, the updated linear layer, the updated audio cue vector, and the updated text cue vector are used to form the trained emotion classification model.

[0102] In some embodiments, the above-mentioned updating of the audio cue vector, text cue vector, and linear layer based on the error between the emotion classification result and the pre-labeled label can be achieved in the following way: substituting the emotion classification result and the pre-labeled label into the loss function to obtain the corresponding error; fixing the parameters of the pre-trained converter model, and updating the parameters of the linear layer, as well as the audio cue vector and text cue vector based on the error.

[0103] Here, the loss function can be the cross-entropy loss function, the binary cross-entropy loss function, or the root mean square error loss function. The specific loss function can be determined according to the needs of the model, and no specific limitation is made here.

[0104] In some embodiments, the model's prediction results (emotion classification results) are compared with pre-labeled labels, and a loss function (such as cross-entropy loss, mean squared error, etc.) is used to calculate the difference between the two results to obtain the corresponding error; the parameters of the pre-trained converter model are fixed, backpropagation is performed based on the obtained error, and the parameters of the linear layer (first parameter and second parameter), as well as the audio cue vector and text cue vector are updated during the backpropagation process.

[0105] It should be noted that this process may require multiple iterations and may require adjusting hyperparameters (such as learning rate, batch size, number of iterations, etc.) to optimize model performance. In addition, the computational efficiency and accuracy of backpropagation can be improved by using various techniques and optimization methods, such as gradient checking, batch normalization, and weight initialization strategies.

[0106] In some embodiments, the above-described error-based update of the parameters of the linear layer, as well as the audio cue vector and the text cue vector, can be achieved by: performing backpropagation based on the error, determining the gradients of the linear layer parameters, the audio cue vector, and the text cue vector during the backpropagation process; and updating the linear layer parameters, the audio cue vector, and the text cue vector based on the gradients.

[0107] Here, the audio cue vector and the text cue vector are treated as a special vector layer, which are special parameters that can be regarded as special weights in the emotion classification model.

[0108] For example, backpropagation is performed based on the calculated error. During backpropagation, the gradients of the first and second parameters in the linear layer are first calculated, and the first and second parameters in the linear layer are updated based on these gradients. Next, the gradients of the audio cue vector and the text cue vector are calculated separately, and the audio cue vector and text cue vector are updated based on these gradients. Through this process, while keeping the pre-trained converter model parameters unchanged, the model can be adapted to a new emotion classification task by fine-tuning only the parameters of the linear layer, the audio cue vector, and the text cue vector. This reduces the number of parameters the model needs to train, lowers the training cost, and allows for higher accuracy with less training data.

[0109] The emotion classification method provided in the embodiments of this application will be further described below.

[0110] In some embodiments, for an object whose emotion needs to be classified, an emotion classification model is invoked to classify it based on the object's text feature vector and audio feature vector to obtain the object's emotion classification result. The emotion classification model is trained according to the emotion classification model training method.

[0111] For example, suppose on a social media platform, users express their emotions through text and voice comments. A user posts a text comment: "This product is terrible, I will never buy it again!" Similarly, the user posts a voice comment. Next, word embedding technology is used to convert the text comment into a text feature vector, and audio processing technology is used to extract the audio feature vector from the voice comment. Subsequently, the text feature vector and audio feature vector are input into a trained emotion classification model for classification. The model outputs an emotion classification result (e.g., "negative emotion") based on the input multimodal feature vectors.

[0112] This application employs a prompt-tuning model training method. By freezing all parameters of the pre-trained model, only the prompt word vectors are trained, resulting in an exponential decrease in the number of parameters to be trained and better avoiding the overfitting problem caused by fine-tuning. Since both speech and text modalities are trained using prompt-tuning, the overfitting-underfitting problem between modalities caused by the inconsistent fitting difficulty between the two modalities during fine-tuning can be avoided, resulting in a higher accuracy model compared to fine-tuning.

[0113] Secondly, in telemarketing and customer service scenarios, labeling multimodal data is challenging and costly, and fine-tuning training methods require a large amount of training data. However, prompt-tuning training methods can achieve better prediction results with less data, making them more suitable for real-world business needs. For example, in a telemarketing scenario, only 500 training data points are needed to achieve a model classification accuracy exceeding 94%, fully meeting business requirements. Furthermore, in telemarketing scenarios, the emotion classification model obtained using the training method provided in this application achieves an accuracy of over 95% in classifying customer emotions (negative and non-negative emotions) based on text and audio.

[0114] Furthermore, see Table 1, which is a comparison table of the performance of the emotion classification model of this application and related technologies. According to Table 1, when the model is trained using the internationally authoritative public dataset IEMOCAP for multi-emotion classification, the emotion classification model provided by this application achieves the best classification performance.

[0115] Table 1. Comparison of the Emotion Classification Model of this Application with Related Technologies

[0116] Model Accuracy ACC Multimodal emotion recognition model MMER 81.7％ Domain Adversarial Neural Network Model (DANN) 82.7％ This application 83.1％

[0117] The following describes an exemplary application of the embodiments of this application in a real-world scenario. This exemplary application describes the training method of the emotion classification model and the specific implementation process of the emotion classification method in a telemarketing scenario.

[0118] See Figure 6 , Figure 6 This is a schematic diagram illustrating the principle of the training method for the emotion classification model provided in this application embodiment. The emotion classification model includes a transformation layer and a linear layer, which will combine... Figure 6 The training method for the emotion classification model and the emotion classification method are explained in detail.

[0119] During the training phase of the emotion classification model, the audio information of the sample objects is extracted using a one-dimensional convolutional neural network (CNN1D) to obtain audio features. These audio features are then embedded to obtain audio feature vectors, which are subsequently input into the speech modality conversion layer. Similarly, the text information of the sample objects is embedded using a tokenizer to obtain text feature vectors, which are then input into the text modality conversion layer.

[0120] Here, the self-attention algorithm is used in the transformation layer; see [link to documentation]. Figure 6 The audio modality conversion layer consists of audio cue vectors and a pre-trained converter model, while the text modality conversion layer consists of text cue vectors and a pre-trained converter model. During model training, the parameters of the pre-trained converter model in the conversion layer are completely frozen, and only the audio cue vectors and text cue vectors need to be updated.

[0121] The self-attention mechanism in the traditional transition layer can be represented by formulas (1)-(4).

[0122] Q = X·W Q (1)

[0123] K = X·W K (2)

[0124] V = X·W V (3)

[0125]

[0126] Where Q is the query vector, K is the key vector, V is the value vector, X is the embedding representation of the input sequence, and W... Q W K and W V These are the weight matrices for the query, key, and value, respectively, where dk is the dimension of the key vector.

[0127] This application adopts the prompt-tuning training method, which adds trainable prompt vector weights before the key vector K and the value vector V respectively. The modified self-attention part can be represented by formulas (5)-(9).

[0128] P_X = cat(P,X) (5)

[0129] Q = X·W Q (6)

[0130] P_K=P_X·W K (7)

[0131] P_V=P_X·W V (8)

[0132]

[0133] Where P_X is the trainable cue word vector weight added before the sum of values ​​in the self-attention of the two modalities, P is the cue vector that needs to be adjusted, X is the embedding representation of the input sequence, P_K is the key vector of the modified self-attention mechanism, and P_V is the value vector in the modified self-attention mechanism.

[0134] After the above self-attention mechanism is used for calculation, the converted audio feature vector and the converted text feature vector are obtained. The converted audio feature vector and the converted text feature vector are input into the linear layer. The features of multiple dimensions can be combined by formula (10). Then, the score of emotion classification is calculated by softmax formula (11). The emotion is classified according to the score to obtain the emotion classification result. The emotion classification result and the pre-labeled label are substituted into the loss function to obtain the corresponding error. The parameters of the pre-trained converter model are fixed, and the parameters of the linear layer, the audio cue vector and the text cue vector are updated based on the error.

[0135] Here, the linear layer includes trainable first and second parameters.

[0136] x=cat(Audio_Encoder(input_audio),Text_Encoder(input_text)) (10)

[0138] Emo_Score=softmax(wx+b) (11)

[0139] In this module, Audio_Encoder represents the overall speech feature extractor module, Text_Encoder represents the overall text extractor module, and the final output sentiment score is Emo_score. w is the weight of the model in the linear layer, and b is the bias of the model in the linear layer. Both w and b are obtained through model training.

[0140] In the emotion classification stage, for the object that needs to be classified by emotion, the emotion classification model is called to classify it based on the object's text feature vector and audio feature vector, and the emotion classification result of the object is obtained.

[0141] The following description continues to illustrate the exemplary structure of the training device 543 for the emotion classification model provided in this application embodiment as a software module. In some embodiments, such as... Figure 2A As shown, the software modules stored in the training device 543 of the emotion classification model in the memory 540 may include: an addition module 5431, a conversion module 5432, a prediction module 5433, and an update module 5434.

[0142] The module 5431 is used to add audio cue vectors and text cue vectors to the transformation layer of the emotion classification model; the transformation module 5432 is used to transform the audio feature vector of the sample object through the transformation layer to obtain the transformed audio feature vector, and to transform the text feature vector of the sample object through the transformation layer to obtain the transformed text feature vector; the prediction module 5433 is used to classify the sample object through a linear layer based on the transformed audio feature vector and the transformed text feature vector to obtain the emotion classification result of the sample object; the update module 5434 is used to update the linear layer, audio cue vector, and text cue vector based on the error between the emotion classification result and the pre-labeled label.

[0143] In some embodiments, the conversion layer includes a first attention mechanism and a second attention mechanism. The conversion module 5432 is further configured to convert the audio feature vector through the first attention mechanism to obtain the converted audio feature vector; and to convert the text feature vector through the second attention mechanism to obtain the converted text feature vector.

[0144] In some embodiments, the conversion module 5432 is further configured to concatenate the audio cue vector and the audio feature vector to obtain a first concatenated vector; determine the first query vector, the first key vector, and the first value vector of the first attention mechanism based on the first concatenated vector and the audio feature vector; and determine the converted audio feature vector based on the first query vector, the first key vector, and the first value vector.

[0145] In some embodiments, the conversion module 5432 is further configured to multiply the audio feature vector with the first weight matrix and use the first multiplication result as the first query vector; multiply the first concatenated vector with the second weight matrix and use the second multiplication result as the first key vector; and multiply the first concatenated vector with the third weight matrix and use the third multiplication result as the first value vector.

[0146] In some embodiments, the conversion module 5432 is further configured to transpose the first key vector to obtain a transposed first key vector; multiply the transposed first key vector with the first query vector to obtain a fourth multiplication result; divide the fourth multiplication result by the square root of the dimension of the first key vector to obtain a first division result; normalize the first division result to obtain a normalized first division result; multiply the normalized first division result with the first value vector, and use the resulting fifth multiplication result as the converted audio feature vector.

[0147] In some embodiments, the conversion module 5432 is further configured to concatenate the text prompt vector and the text feature vector to obtain a second concatenated vector; determine the second query vector, the second key vector and the second value vector of the second attention mechanism based on the second concatenated vector and the text feature vector; and determine the converted text feature vector based on the second query vector, the second key vector and the second value vector.

[0148] In some embodiments, the conversion module 5432 is further configured to multiply the text feature vector with the fourth weight matrix and use the resulting sixth multiplication result as the second query vector; multiply the second concatenated vector with the fifth weight matrix and use the resulting seventh multiplication result as the second key vector; and multiply the second concatenated vector with the sixth weight matrix and use the resulting eighth multiplication result as the second value vector.

[0149] In some embodiments, the conversion module 5432 is further configured to transpose the second key vector to obtain a transposed second key vector; multiply the transposed second key vector with the second query vector to obtain a ninth multiplication result; divide the ninth multiplication result by the square root of the dimension of the second key vector to obtain a second division result; normalize the second division result to obtain a normalized second division result; multiply the normalized second division result with the second value vector, and use the resulting tenth multiplication result as the converted text feature vector.

[0150] In some embodiments, the conversion layer further includes a pre-trained converter model, and the update module 5434 is also used to substitute the emotion classification result and the pre-labeled label into the loss function to obtain the corresponding error; fix the parameters of the pre-trained converter model, and update the parameters of the linear layer, the audio cue vector and the text cue vector based on the error.

[0151] In some embodiments, the update module 5434 is further configured to perform backpropagation based on the error, and determine the gradients of the linear layer parameters, the audio cue vector, and the text cue vector respectively during the backpropagation process; and update the linear layer parameters, the audio cue vector, and the text cue vector respectively based on the gradients.

[0152] In some embodiments, such as Figure 2B As shown, the software modules stored in the emotion classification device 643 in the memory 640 may include: classification module 6431.

[0153] The classification module 6431 is used to classify objects whose emotions are to be classified by calling an emotion classification model based on the text feature vector and audio feature vector of the object, and to obtain the emotion classification result of the object. The emotion classification model is trained according to the training method of the emotion classification model provided in the embodiments of this application.

[0154] It should be noted that the description of the apparatus in this application embodiment is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, therefore it will not be repeated. For any technical details not covered in the training apparatus for the emotion classification model provided in this application embodiment, please refer to... Figure 3 , Figure 4 ,or Figure 5 The meaning is understood in accordance with the description of any of the accompanying drawings.

[0155] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. The processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the training method or emotion classification method of the emotion classification model described above in this application.

[0156] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the training method or emotion classification method of the emotion classification model provided in this application. For example, ... Figure 3 , Figure 4 ,or Figure 5 The training method for the emotion classification model is shown.

[0157] In some embodiments, the computer-readable storage medium may be a memory such as ferroelectric random access memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); or it may be a device that includes one or any combination of the above-mentioned memories.

[0158] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0159] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0160] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0161] In summary, the embodiments of this application have the following beneficial effects:

[0162] (1) The prompt-tuning module based on speech modality enables better utilization of audio information and enhances the capabilities of multimodal models;

[0163] (2) In the multimodal framework, both text modality and speech modality are trained through prompt-tuning. The parameters of the pre-trained model are frozen during training, which avoids the problem of inconsistent fitting degree between modalities caused by fine-tuning. This makes the fitting ability of each module of the multimodal model reach a better balance, thereby improving the overall model performance.

[0164] (3) It has a generally better effect in small sample learning and can effectively reduce the model training cost. Therefore, it has a greater advantage in fields where the cost of multimodal labeling is high.

[0165] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A training method for an emotion classification model, characterized in that, The method includes: Add audio cue vectors and text cue vectors to the transformation layer of the emotion classification model; The audio feature vector of the sample object is transformed by the transformation layer to obtain the transformed audio feature vector, and the text feature vector of the sample object is transformed by the transformation layer to obtain the transformed text feature vector. Based on the converted audio feature vector and the converted text feature vector, the emotion classification model is used to classify the sample object through a linear layer to obtain the emotion classification result. Based on the error between the emotion classification result and the pre-labeled tags, the audio cue vector, the text cue vector, and the linear layer are updated.

2. The method according to claim 1, characterized in that, The conversion layer includes a first attention mechanism and a second attention mechanism; The process of converting the audio feature vector of the sample object through the conversion layer to obtain the converted audio feature vector, and converting the text feature vector of the sample object through the conversion layer to obtain the converted text feature vector, includes: The audio feature vector is transformed using the first attention mechanism to obtain the transformed audio feature vector; The text feature vector is transformed using the second attention mechanism to obtain the transformed text feature vector.

3. The method according to claim 2, characterized in that, The step of transforming the audio feature vector through the first attention mechanism to obtain the transformed audio feature vector includes: The audio cue vector and the audio feature vector are concatenated to obtain a first concatenated vector; Based on the first concatenation vector and the audio feature vector, the first query vector, the first key vector, and the first value vector of the first attention mechanism are determined. Based on the first query vector, the first key vector, and the first value vector, the converted audio feature vector is determined.

4. The method according to claim 3, characterized in that, The step of determining the first query vector, first key vector, and first value vector of the first attention mechanism based on the first concatenation vector and the audio feature vector includes: The audio feature vector is multiplied by the first weight matrix, and the result of the first multiplication is used as the first query vector. Multiply the first concatenated vector with the second weight matrix, and use the result of the second multiplication as the first key vector; Multiply the first concatenated vector with the third weight matrix, and use the result of the third multiplication as the first value vector.

5. The method according to claim 3, characterized in that, The step of determining the converted audio feature vector based on the first query vector, the first key vector, and the first value vector includes: Transpose the first key vector to obtain the transposed first key vector; The transposed first key vector is multiplied by the first query vector to obtain the fourth multiplication result; Divide the fourth multiplication result by the square root of the dimension of the first key vector to obtain the first division result; The first division result is normalized to obtain the normalized first division result; The normalized first division result is multiplied by the first value vector, and the resulting fifth multiplication result is used as the converted audio feature vector.

6. The method according to claim 2, characterized in that, The step of transforming the text feature vector through the second attention mechanism to obtain the transformed text feature vector includes: The text prompt vector and the text feature vector are concatenated to obtain a second concatenated vector; Based on the second concatenation vector and the text feature vector, the second query vector, the second key vector, and the second value vector of the second attention mechanism are determined. Based on the second query vector, the second key vector, and the second value vector, the transformed text feature vector is determined.

7. The method according to claim 6, characterized in that, The step of determining the second query vector, second key vector, and second value vector of the second attention mechanism based on the second concatenation vector and the text feature vector includes: The text feature vector is multiplied by the fourth weight matrix, and the result of the sixth multiplication is used as the second query vector. Multiply the second concatenated vector with the fifth weight matrix, and use the resulting seventh multiplication result as the second key vector; Multiply the second concatenated vector with the sixth weight matrix, and use the resulting eighth multiplication result as the second value vector.

8. The method according to claim 6, characterized in that, The step of determining the transformed text feature vector based on the second query vector, the second key vector, and the second value vector includes: Transpose the second key vector to obtain the transposed second key vector; Multiply the transposed second key vector with the second query vector to obtain the ninth multiplication result; Divide the result of the ninth multiplication by the square root of the dimension of the second key vector to obtain the second division result; The second division result is normalized to obtain the normalized second division result; The normalized second division result is multiplied by the second value vector, and the resulting tenth multiplication result is used as the transformed text feature vector.

9. The method according to any one of claims 1 to 8, characterized in that, The conversion layer also includes a pre-trained converter model; The step of updating the audio cue vector, the text cue vector, and the linear layer based on the error between the emotion classification result and the pre-labeled tags includes: Substitute the emotion classification results and pre-labeled tags into the loss function to obtain the corresponding error; The parameters of the pre-trained converter model are fixed, and the parameters of the linear layer, as well as the audio cue vector and the text cue vector, are updated based on the error.

10. The method according to claim 9, characterized in that, The step of updating the parameters of the linear layer, the audio cue vector, and the text cue vector based on the error includes: Backpropagation is performed based on the error, and the gradients of the parameters of the linear layer, the gradient of the audio cue vector, and the gradient of the text cue vector are determined during the backpropagation process. The parameters of the linear layer, the audio cue vector, and the text cue vector are updated based on the gradient.

11. An emotion classification method, characterized in that, The method includes: For an object whose emotion needs to be classified, an emotion classification model is invoked based on the text feature vector and audio feature vector of the object to perform classification, thereby obtaining the emotion classification result of the object, wherein the emotion classification model is trained by the method according to any one of claims 1 to 10.

12. A training device for an emotion classification model, characterized in that, The device includes: Add a module to add audio cue vectors and text cue vectors to the transformation layer of the emotion classification model; The conversion module is used to convert the audio feature vector of the sample object through the conversion layer to obtain the converted audio feature vector, and to convert the text feature vector of the sample object through the conversion layer to obtain the converted text feature vector; The prediction module is used to classify the sample object based on the converted audio feature vector and the converted text feature vector through the linear layer included in the emotion classification model, so as to obtain the emotion classification result of the sample object. An update module is used to update the audio cue vector, the text cue vector, and the linear layer based on the error between the emotion classification result and the pre-labeled label.

13. An emotion classification device, characterized in that, The device includes: The classification module is used to classify an object whose emotion needs to be classified by calling an emotion classification model based on the text feature vector and audio feature vector of the object, and to obtain the emotion classification result of the object, wherein the emotion classification model is trained by the method according to any one of claims 1 to 10.

14. An electronic device, characterized in that, include: Memory is used to store executable instructions or computer programs. The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the training method of the emotion classification model according to any one of claims 1 to 10, or implements the emotion classification method according to claim 11.

15. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by the processor, they implement the training method of the emotion classification model according to any one of claims 1 to 10, or the emotion classification method according to claim 11.

16. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by the processor, they implement the training method of the emotion classification model according to any one of claims 1 to 10, or the emotion classification method according to claim 11.

Images