Emotion recognition model training method, emotion recognition method and electronic equipment

Abstract

The invention discloses an emotion recognition model training method, an emotion recognition method and electronic equipment, and the method comprises the steps: determining a training data set which comprises a plurality of emotion data samples, the plurality of sentiment data samples comprise single-mode sentiment data samples obtained according to one of image sentiment information, sound sentiment information and text sentiment information, and multi-mode sentiment data samples obtained according to at least two of the image sentiment information, the sound sentiment information and the text sentiment information; and performing model training on the initial emotion recognition model according to the training data set to obtain an emotion recognition model. Therefore, the emotion recognition model can perform emotion recognition according to the emotion information of any mode, and emotion recognition is more comprehensive. And one emotion recognition model can realize recognition processing of at least one type of modal information in the image, the sound and the text, so that the emotion recognition efficiency is higher.

Description

Technical Field

[0001] The present application relates to the field of emotion recognition technology, and in particular to an emotion recognition model training method, an emotion recognition method and an electronic device. Background Art

[0002] Emotions can be expressed through a variety of media, such as expressions, sounds, and text. With the development of computer technology, the application of technologies that intelligently identify user emotions based on trained emotion recognition models, obtain the emotional attributes of user emotions such as happiness and anger, and then interact with users based on the identified user emotions is becoming more and more widespread. Therefore, the recognition accuracy, efficiency, and other recognition effects of emotion recognition models directly affect the user experience. How to train emotion recognition models with better recognition effects is a problem that everyone in the field is concerned about. Summary of the invention

[0003] The present application provides an emotion recognition model training method, an emotion recognition method and an electronic device, wherein the emotion recognition model is trained based on single-modal emotion data including one of image emotion information, sound emotion information and text emotion information, and multi-modal emotion data including at least two of image emotion information, sound emotion information and text emotion information, so that the trained emotion recognition model can perform emotion recognition based on emotion data of any modality, making the emotion recognition more comprehensive. Moreover, an emotion recognition model can realize the recognition processing of at least one input information of image, sound and text, and obtain the emotion recognition result, making the emotion recognition faster and more efficient.

[0004] To solve the above technical problems, in the first aspect, the implementation method of the present application discloses a method for training an emotion recognition model, which method includes: determining a training data set, the training data set includes multiple emotion data samples, the multiple emotion data samples are obtained based on image emotion information, sound emotion information, and text emotion information, and the multiple emotion data samples include multiple unimodal emotion data samples and multiple multimodal emotion data samples, each unimodal emotion data sample includes one of the image emotion information, sound emotion information, and text emotion information, and each multimodal emotion data sample includes at least two of the image emotion information, sound emotion information, and text emotion information; performing model training on an initial emotion recognition model according to the training data set to obtain an emotion recognition model.

[0005] The emotion recognition model training method provided by the implementation of the present application determines a single-modal emotion data sample including one of the image emotion information, sound emotion information, and text emotion information, and a multi-modal emotion data sample including at least two of the image emotion information, sound emotion information, and text emotion information to perform model training on the initial emotion recognition model to obtain a trained emotion recognition model, so that the trained emotion recognition model can perform emotion recognition based on emotion data of any modality, making emotion recognition more comprehensive. Moreover, an emotion recognition model can realize the recognition and processing of at least one input information of image, sound, and text to obtain an emotion recognition result, making the emotion recognition faster and more efficient.

[0006] According to another specific implementation of the present application, the emotion recognition model training method disclosed in the implementation of the present application, the emotion data sample is obtained in the following manner: acquiring emotion information, the emotion information includes image emotion information, sound emotion information and text emotion information; performing data enhancement processing on the emotion information to obtain the emotion data sample.

[0007] According to another specific implementation of the present application, the emotion recognition model training method disclosed in the implementation of the present application obtains emotion information, including: obtaining emotion video information; performing information extraction processing on the emotion video information to obtain emotion information.

[0008] By adopting the above technical solution, the emotional video information is extracted and processed to obtain emotional information including image emotional information, sound emotional information and text emotional information. In this way, the initial emotion recognition model can be trained based on one of the image emotional information, sound emotional information and text emotional information, or multimodal emotional data samples of at least two types of information to obtain a trained emotion recognition model, so that the trained emotion recognition model can perform emotion recognition according to any modality of emotional data, making the emotion recognition more comprehensive.

[0009] According to another specific implementation of the present application, the emotion recognition model training method disclosed in the implementation of the present application performs data enhancement processing on emotion information to obtain emotion data samples, including: performing data enhancement processing on image emotion information to obtain image data; performing data enhancement processing on sound emotion information to obtain sound data; performing data enhancement processing on text emotion information to obtain text data; performing data enhancement processing on image emotion information and sound emotion information to obtain image and sound data; performing data enhancement processing on image emotion information and text emotion information to obtain image and text data; performing data enhancement processing on image emotion information, sound emotion information and text emotion information to obtain image and sound text data; obtaining emotion data samples according to image data, sound data, text data, image and sound data, image and text data, sound and text data, and image and sound text data.

[0010] By adopting the above technical scheme, data enhancement processing is performed on the emotion information, and seven emotion data samples containing different modalities can be obtained based on one emotion information. In this way, the model is trained based on the seven emotion data samples, so that the emotion recognition model can perform normal emotion recognition processing when receiving any single-modal or multi-modal emotion feature data to obtain emotion recognition results, thereby improving the emotion recognition accuracy of the emotion recognition model and improving the emotion recognition efficiency.

[0011] According to another specific implementation of the present application, the emotion recognition model training method disclosed in the implementation of the present application performs information extraction processing on emotion video information to obtain emotion information, including: performing image extraction processing on emotion video information through an image extraction algorithm to obtain image emotion information; and performing sound extraction processing on emotion video information through a sound extraction algorithm to obtain sound emotion information; and performing text extraction processing on sound emotion information through a text recognition algorithm to obtain text emotion information.

[0012] According to another specific implementation of the present application, the emotion recognition model training method disclosed in the implementation of the present application performs model training on the initial emotion recognition model according to the training data set to obtain the emotion recognition model, including: performing feature extraction processing on the emotion data samples to obtain emotion feature data, and performing model training on the initial emotion recognition model according to the emotion feature data to obtain the emotion recognition model.

[0013] By adopting the above technical solution, seven emotion data samples containing different modalities can be obtained based on one emotion information, and then the model is trained based on the seven emotion data samples, so that the emotion recognition model can perform normal emotion recognition processing when receiving any single-modal or multi-modal emotion feature data to obtain emotion recognition results, thereby improving the emotion recognition accuracy of the emotion recognition model and improving the emotion recognition efficiency.

[0014] According to another specific implementation of the present application, the emotion recognition model training method disclosed in the implementation of the present application performs feature extraction processing on emotion data samples to obtain emotion feature data, including: performing facial recognition processing on emotion data samples including image emotion information based on a facial recognition model to obtain a facial image, and performing feature extraction processing on the facial image based on a visual feature extractor to obtain image feature data; performing sound feature extraction processing on emotion data samples including sound emotion information based on a sound feature extractor to obtain sound feature data; performing text feature extraction processing on emotion data samples including text emotion information based on a text feature extractor to obtain text feature data; using image feature data, sound feature data and text feature data as emotion feature data.

[0015] In the second aspect, the implementation method of the present application also discloses an emotion recognition method, which includes: determining the user's emotion data, performing feature extraction processing on the emotion data to obtain emotion feature data, inputting the emotion feature data into an emotion recognition model, and obtaining the user's emotion recognition result, and the emotion recognition model is obtained by model training based on the emotion recognition model training method provided by any one of the implementation methods provided in the first aspect above.

[0016] According to another specific implementation of the present application, the emotion recognition method disclosed in the implementation of the present application inputs emotion feature data into an emotion recognition model to obtain an emotion recognition result of the user, including: inputting the emotion feature data into a classifier of the emotion recognition model so that the classifier determines the emotion category according to the emotion feature data as the emotion recognition result.

[0017] In the emotion recognition method provided by the implementation of the present application, the emotion feature data can be unimodal emotion feature data including one of the modal data of image feature data, sound feature data, and text feature data, or can be multimodal emotion feature data including at least two of the modal data of image feature data, sound feature data, and text feature data. In this way, in the absence of any one of the modalities, the emotion recognition model can perform emotion recognition and obtain the user emotion category (as an example of the emotion recognition result). In this way, emotion feature data of different modalities can be received for emotion recognition processing based on only one emotion recognition model, which reduces the cost of emotion recognition and speeds up the efficiency of emotion recognition.

[0018] In a third aspect, an implementation of the present application provides an electronic device, comprising: a processor, and a memory communicatively connected to the processor, the processor comprising a central processing unit and a graphics processing unit; the memory stores a computer program, and the processor executes the computer program stored in the memory, so that the electronic device implements the emotion recognition model training method provided in any one of the implementations of the first aspect above, and / or the emotion recognition method provided in any one of the implementations of the second aspect above.

[0019] In a fourth aspect, an implementation of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the emotion recognition model training method provided in any one of the implementations of the first aspect above, and / or the emotion recognition method provided in any one of the implementations of the second aspect above.

[0020] In a fifth aspect, an implementation method of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the emotion recognition model training method provided in any one of the implementation methods of the first aspect above, and / or the emotion recognition method provided in any one of the implementation methods of the second aspect above.

[0021] It can be understood that the beneficial effects of the third to fifth aspects mentioned above can also be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a flowchart of an emotion recognition model training method provided in an embodiment of the present application;

[0023] Figure 2 It is a flowchart of an emotion recognition method provided in an embodiment of the present application;

[0024] Figure 3 It is a schematic diagram of the principle of training an emotion recognition model for emotion recognition provided in an embodiment of the present application;

[0025] Figure 4 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] As mentioned above, emotions can be expressed through a variety of media, such as facial expressions, sounds, text, or physiological signals such as heart rate and EEG. Due to the limitations of physiological signal acquisition equipment, most current electronic devices such as mobile phones and computers obtain user video image information or voice, text and other information based on trained emotion recognition models (i.e., emotion perception systems) to perform intelligent recognition of user emotions, in order to obtain the user's current emotional attributes such as happiness and anger, and to interact with the user based on the recognized user emotions, so that AI can be more "humanized". Therefore, the recognition accuracy, efficiency and other recognition effects of the emotion recognition model directly affect the user experience. How to train an emotion recognition model with better recognition effect is a problem that everyone in the field is concerned about.

[0027] The current emotion perception model (i.e., emotion recognition model) only supports the input of one emotion modality or a fixed modality combination, such as emotion recognition only through expression images or emotion recognition through a combination of expression and text. This is because when the emotion perception model is currently trained, most of the corresponding emotion recognition models are trained based on any one of the feature data of the three modalities of expression images, sound, and text, or any two combined feature data. For example, the model is trained according to expression images to obtain an image emotion recognition model that can recognize images, the model is trained according to sounds to obtain a sound emotion recognition model that recognizes sounds, the model is trained according to text to obtain a text emotion recognition model that recognizes text, and the model is trained according to expression images and sounds to obtain an emotion recognition model that can recognize image + sound combinations, etc. In this way, multiple models of different dimensions can be generated. This training method can only receive the corresponding feature information for the corresponding emotion recognition model. If it is other feature information, the emotion recognition model cannot perform emotion recognition, and an error will be generated, which has limitations. However, in real applications, users may only have changes in facial expressions but no sound, or they may speak but their facial expressions do not change. In these cases, it is difficult to handle them using only the same model. However, using multiple models of different dimensions will reduce the speed of emotion recognition and reduce recognition efficiency. In addition, multiple models will take up more resources and are difficult to be applied in real life.

[0028] Based on this, the implementation of the present application provides an emotion recognition model training method and an emotion recognition method, which trains the emotion recognition model based on a single-modal emotion data sample including one of the image emotion information, sound emotion information, and text emotion information, and a multi-modal emotion data sample including at least two of the image emotion information, sound emotion information, and text emotion information, so that the trained emotion recognition model can perform emotion recognition based on emotion data of any modality, making emotion recognition more comprehensive. Moreover, an emotion recognition model can realize the recognition and processing of at least one input information of image, sound, and text to obtain an emotion recognition result, making the emotion recognition faster and more efficient.

[0029] Next, see Figure 1 The emotion recognition model training method provided by the implementation of this application includes the following steps.

[0030] S110, determine a training data set, the training data set includes multiple emotion data samples, the multiple emotion data samples are obtained based on image emotion information, sound emotion information, and text emotion information, and the multiple emotion data samples include multiple unimodal emotion data samples and multiple multimodal emotion data samples, each unimodal emotion data sample includes one of the image emotion information, sound emotion information, and text emotion information, and each multimodal emotion data sample includes at least two of the image emotion information, sound emotion information, and text emotion information.

[0031] S120, performing model training on the initial emotion recognition model according to the training data set to obtain an emotion recognition model.

[0032] The emotion recognition model training method provided by the implementation of the present application determines a single-modal emotion data sample including one of the image emotion information, the sound emotion information, and the text emotion information, and a multi-modal emotion data sample including at least two of the image emotion information, the sound emotion information, and the text emotion information to perform model training on the initial emotion recognition model to obtain a trained emotion recognition model, so that the trained emotion recognition model can perform emotion recognition based on emotion data of any modality, making emotion recognition more comprehensive. Moreover, an emotion recognition model can realize the recognition and processing of at least one input information of image, sound, and text to obtain an emotion recognition result, making the emotion recognition faster and more efficient.

[0033] In the implementation of the present application, the emotion data sample is obtained in the following manner: acquiring emotion information, the emotion information including image emotion information, sound emotion information and text emotion information; performing data enhancement processing on the emotion information to obtain the emotion data sample.

[0034] Exemplarily, in one implementation of the present application, acquiring emotion information includes: acquiring image emotion information (V), sound emotion information (A), and text emotion information (T) that are input separately.

[0035] In another implementation of the present application, obtaining emotional information includes: obtaining input emotional video information containing image emotional information, sound emotional information, and text emotional information.

[0036] Furthermore, the emotional video information is subjected to information extraction processing to obtain emotional information.

[0037] In the implementation of the present application, information extraction processing is performed on emotional video information to obtain emotional information, including: image extraction processing is performed on video information through an image extraction algorithm to obtain image emotional information.

[0038] Exemplarily, when the input emotional information is emotional video information, the emotional video information needs to be preprocessed.

[0039] Specifically, video editing software such as ffmpeg (as an example of an image extraction algorithm) is used to divide the video into frames to obtain image emotion information.

[0040] And the sound extraction algorithm is used to process the video information to obtain the sound emotion information.

[0041] Specifically, video editing software such as ffmpeg (as an example of a sound extraction algorithm) is used to extract audio information from emotional video information to obtain sound emotional information.

[0042] Among them, ffmpeg is an audio and video processing software, based on the ffmpeg command line tool, video editing and audio extraction can be performed. Of course, the implementation of this application can also use other image extraction algorithms for image extraction and other sound extraction algorithms for sound extraction.

[0043] And the sound emotion information is processed by text extraction through a text recognition algorithm to obtain text emotion information.

[0044] Specifically, an ASR algorithm (as an example of a text recognition algorithm) is used to perform text extraction processing on the text in the sound emotion information to obtain text emotion information.

[0045] Among them, the ASR algorithm is a speech-to-text algorithm, which can convert the sound in the audio into text. Of course, the implementation of this application can also use other text extraction algorithms to extract text.

[0046] In this way, emotional information in three modalities, image (V), sound (A), and text (T), can be obtained.

[0047] Furthermore, after extracting the image emotion information, the sound emotion information and the text emotion information, it is necessary to perform data enhancement processing on the emotion information to obtain emotion data samples.

[0048] Exemplarily, during the training of the emotion recognition model, the emotion information is subjected to data enhancement processing, that is, the modality is expanded, and one piece of emotion data containing three modalities, namely, image emotion information (V), sound emotion information (A), and text emotion information (T), is expanded to 7 pieces of emotion data.

[0049] In the implementation of this application, the emotional information is subjected to data enhancement processing, and the emotional data samples obtained include:

[0050] The image emotional information (V) is subjected to data enhancement processing to obtain image data, which only contains the image emotional information (V). The sound emotional information (A) and the text emotional information (T) are filled with 0 to obtain image data of V+0+0.

[0051] The sound emotion information (A) is subjected to data enhancement processing to obtain sound data, which only contains the sound emotion information (A). The image emotion information (V) and the text emotion information (T) are filled with 0 to obtain image data of 0+A+0.

[0052] The text emotion information (T) is subjected to data enhancement processing to obtain text data, which only contains the text emotion information (T). The image emotion information (V) and the sound emotion information (A) are filled with 0 to obtain 0+0+T text data.

[0053] Data enhancement processing is performed on the image emotion information (V) and the sound emotion information (A) to obtain image and sound data, which includes the image emotion information (V) and the sound emotion information (A). The text emotion information (T) is filled with 0 to obtain image and sound data of V+A+0.

[0054] The image emotion information (V) and the text emotion information (T) are subjected to data enhancement processing to obtain image text data, which includes the image emotion information (V) and the text emotion information (T). The sound emotion information (A) is filled with 0 to obtain V+0+T image text data.

[0055] The sound emotion information (A) and the text emotion information (T) are data enhanced to obtain sound text data, which includes the sound emotion information (A) and the text emotion information (T). The image emotion information (V) is filled with 0 to obtain 0+A+T sound text data.

[0056] Data enhancement processing is performed on the image emotion information (V), the sound emotion information (A) and the text emotion information (T) to obtain image sound text data, which includes the image emotion information (V), the sound emotion information (A) and the text emotion information (T) to obtain image sound text data of V+A+T.

[0057] In this way, the emotion data sample is obtained according to the image data (V+0+0), sound data (0+A+0), text data (0+0+T), image and sound data (V+A+0), image and text data (V+0+T), sound and text data (0+A+T), and image and sound and text data (V+A+T).

[0058] Furthermore, the initial emotion recognition model is trained according to the training data set to obtain an emotion recognition model.

[0059] In the implementation method of the present application, the initial emotion recognition model is trained according to the training data set to obtain the emotion recognition model, which includes: performing feature extraction processing on the emotion data samples to obtain emotion feature data, and training the initial emotion recognition model according to the emotion feature data to obtain the emotion recognition model.

[0060] Exemplarily, feature extraction processing is performed on the emotion data sample to obtain emotion feature data, including:

[0061] Based on the facial recognition model, facial recognition processing is performed on the emotion data sample including the image emotion information to obtain the facial image, and based on the visual feature extractor, feature extraction processing is performed on the facial image to obtain image feature data.

[0062] For example, taking emotion recognition of people as an example, MTCNN (as an example of a facial recognition model) is used to perform facial recognition processing on the image emotion information included in the emotion data sample, the facial image is identified, and the facial image is input into a visual feature extractor such as VGG for feature extraction processing to obtain image feature data.

[0063] Among them, MTCNN is a face detection algorithm, which can detect faces in images to extract faces. VGG is a convolutional neural network model, which can perform image feature extraction to convert images into feature vectors. Of course, other facial recognition models can also be used in the implementation of this application to perform facial recognition processing, and facial feature extraction processing can be performed based on other visual feature extractors.

[0064] Based on the sound feature extractor, sound feature extraction processing is performed on the emotion data sample including the sound emotion information to obtain sound feature data.

[0065] Exemplarily, a sound feature extractor such as WAV2vec is used to perform feature extraction processing on the sound emotion information included in the emotion data sample to obtain sound feature data.

[0066] Among them, WAV2vec is a speech pre-training model, which can extract sound information and generate sound feature vectors. Of course, other sound feature extractors can also be used in the implementation of this application to perform sound feature extraction processing.

[0067] Based on the text feature extractor, text feature extraction processing is performed on the emotion data sample including the text emotion information to obtain text feature data.

[0068] Exemplarily, a text feature extractor such as Bert is used to perform text feature extraction processing on the text sentiment information in the sentiment data sample to obtain text feature data.

[0069] Among them, the BERT model is a natural language processing model, which can identify text information to perform text feature extraction processing and obtain a text feature vector. Of course, other text feature extractors can also be used in the implementation of this application to perform text feature extraction processing.

[0070] Among them, the image feature data, the sound feature data and the text feature data are vector data after feature extraction.

[0071] In this way, the image feature data, the sound feature data, and the text feature data are used as the emotion feature data.

[0072] Exemplarily, if it is image data (V+0+0), then either the sound feature extraction process and the text feature extraction process are not performed, or the sound feature vector and the text feature vector obtained by performing the sound feature extraction process and the text feature extraction process are 0; if it is sound data (0+A+0), then either the image feature extraction process and the text feature extraction process are not performed, or the image feature data and the text feature data obtained by performing the image feature extraction process and the text feature extraction process are 0; if it is text data (0+0+T), then either the image feature extraction process and the sound feature extraction process are not performed, or the image feature data and the sound feature data obtained by performing the image feature extraction process and the sound feature extraction process are 0; if it is image data For example, for sound data (V+A+0), either text feature extraction is not performed or the text feature data obtained by the text feature extraction is 0. For image text data (V+0+T), either sound feature extraction is not performed or the sound feature data obtained by the sound feature extraction is 0. For sound text data (0+A+T), either image feature extraction is not performed or the image feature data obtained by the image feature extraction is 0. If it is image sound text data (V+A+T), image feature extraction, sound feature extraction and text feature extraction are performed on the image sound text data to obtain image feature data, sound feature data and text feature data.

[0073] In this way, after the emotion feature data of the three modalities corresponding to the seven emotion sample data extracted according to one emotion information are successfully extracted, they are stacked, and the emotion feature data and the corresponding emotion category are input into the fully connected layer of the initial emotion recognition model, so that the initial emotion recognition model performs emotion recognition according to the emotion feature data to obtain the emotion recognition result, and compares the emotion recognition result with the input emotion category. If the output emotion recognition result is different from the input emotion category, adaptive adjustment is performed until the output emotion recognition result is the same as the input emotion category to obtain a trained emotion recognition model.

[0074] In this way, for emotional information such as emotional video information, the emotional information is preprocessed to obtain emotional information of three modes: image emotional information, sound emotional information, and text emotional information. Then, the emotional information of each mode is data enhanced to obtain 7 different single-modal or multi-modal emotional information based on the emotional information of the three modes, so as to train the emotion recognition model based on the single-modal and multi-modal emotional information. Therefore, different modal missing conditions are added during the model training process, so that in the case of any modality missing, the emotion recognition model can normally identify which modality is missing, and expand this part of the missing modal data, and then give the predicted emotion recognition result according to the expanded and feature-extracted emotional feature data. Based on the emotion recognition model of the implementation method of the present application, when performing emotion recognition, it can simultaneously process single-modal and multi-modal emotion data obtained by different modal combinations, and predict the emotion category. And only one emotion recognition model can be used to perform emotion recognition in different dimensions, which improves the efficiency of emotion recognition. On six-category data, the prediction accuracy can reach more than 58.7%.

[0075] Furthermore, the implementation of the present application also provides an emotion recognition method, such as Figure 2 As shown, the specific steps include the following steps.

[0076] S210, determining the user's emotion data, and performing feature extraction processing on the emotion data to obtain emotion feature data.

[0077] S220, input the emotion feature data into the emotion recognition model to obtain the user's emotion recognition result.

[0078] In the implementation method of the present application, the emotion feature data is input into the emotion recognition model to obtain the user's emotion recognition result, including: inputting the emotion feature data into the classifier of the emotion recognition model so that the classifier determines the emotion category according to the emotion feature data as the emotion recognition result.

[0079] Exemplarily, the emotion recognition model includes an input receiver, a classifier, and a result outputter. After receiving the emotion information, the input receiver performs feature extraction processing on the emotion information to obtain corresponding emotion feature data, and sends the emotion feature data to the classifier. The classifier performs classification processing based on the emotion feature data to obtain a corresponding emotion category, and sends the emotion category to the result outputter. The result outputter outputs the emotion category as an emotion recognition result to the electronic device.

[0080] In the emotion recognition method provided by the implementation of the present application, the emotion feature data can be unimodal emotion feature data including one of the modal data of image feature data, sound feature data, and text feature data, or can be multimodal emotion feature data including at least two of the modal data of image feature data, sound feature data, and text feature data. In this way, in the absence of any one of the modalities, the emotion recognition model can perform emotion recognition and obtain the user emotion category (as an example of the emotion recognition result). In this way, emotion feature data of different modalities can be received for emotion recognition processing based on only one emotion recognition model, which reduces the cost of emotion recognition and speeds up the efficiency of emotion recognition.

[0081] In another implementation of the present application, Figure 3 As shown, the emotion recognition model training method and emotion recognition method provided by the implementation of the present application specifically include the following process.

[0082] First, emotional video information such as emotional video or recorded video (as an example of emotional information) is received as input, and data preprocessing (frame segmentation, audio extraction) is performed on the emotional video information to obtain image emotional information (image V) and voice emotional information (also known as sound emotional information) (voice A). And perform text extraction on the sound emotional information (for example, perform speech-to-text processing based on TTS (Text To Speech)) to obtain emotional information such as text emotional information (text T), and take the emotional information such as image V, voice A and text T as input, perform data enhancement processing on the emotional information to obtain emotional data samples such as image data (V), sound data (A), text data (T), image and sound data (V+A), image and text data (V+T), sound and text data (A+T), and image and sound and text data (V+A+T), and then perform image feature extraction processing on the enhanced image V in each emotional data sample based on image feature extractors such as VGG, perform sound feature extraction processing on the enhanced voice A based on sound feature extractors such as WAV2vec, and perform text feature extraction processing on the enhanced text T based on text feature extractors such as bert to obtain emotional feature data such as image feature data, sound feature data and text feature data, and input the emotional feature data into the classifier of the emotion recognition model for model training to obtain the emotional category. In this way, when performing emotion recognition, the emotion recognition model receives emotion information, preprocesses the emotion information, and obtains emotion feature data after feature extraction, and then inputs the emotion feature data into the classifier to obtain the emotion category to output the emotion recognition result.

[0083] The emotion recognition model training method and emotion recognition method provided by the implementation of the present application can be applied to different electronic devices or to the same electronic device.

[0084] The electronic device may specifically be a mobile phone, a vehicle, or other electronic device.

[0085] See also Figure 4 , Figure 4 The figure shows a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Figure 4 As shown, the electronic device may include: a transceiver 121 , a processor 122 , and a memory 123 .

[0086] The processor 122 executes the computer execution instructions stored in the memory, so that the processor 122 executes the emotion recognition model training method in the above embodiment, and / or the technical solution of the emotion recognition method. The processor 122 can be a general-purpose processor, including a central processing unit CPU, a network processor (network processor, NP), etc.; it can also be a digital data processor DSP, an application-specific integrated circuit ASIC, a field programmable gate array FPGA or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0087] The memory 123 is connected to the processor 122 via a system bus and completes communication between them. The memory 123 is used to store computer program instructions.

[0088] By way of example and not limitation, the memory 123 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disk, a magneto-optical disk, a tape, or a universal serial bus (USB) drive, or a combination of two or more of these. Where appropriate, the memory 123 may include a removable or non-removable (or fixed) medium. Where appropriate, the memory 123 may be inside or outside the integrated gateway device. In a particular embodiment, the memory 123 is a non-volatile solid-state memory. In a particular embodiment, the memory 123 includes a read-only memory (ROM). Where appropriate, the ROM may be a mask-programmed ROM, a programmable ROM (PROM), an erasable PROM (EPROM), an electrically erasable PROM (EEPROM), an electrically rewritable ROM (EAROM), or a flash memory, or a combination of two or more of these.

[0089] The transceiver 121 may be used to obtain tasks to be executed and configuration information of the tasks to be executed.

[0090] The system bus can be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The transceiver is used to realize the communication between the database access device and other computers (such as clients, read-write libraries, and read-only libraries). The memory may include random access memory (RAM) and may also include non-volatile memory.

[0091] An embodiment of the present application also provides a chip for running instructions, which is used to execute the emotion recognition model training method in the above embodiment, and / or the technical solution of the emotion recognition method.

[0092] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on a processor of an electronic device, the processor of the electronic device executes the emotion recognition model training method of the above-mentioned embodiment, and / or the technical solution of the emotion recognition method.

[0093] In some possible implementations, various aspects of the method provided in the present application can also be implemented in the form of a program product, which includes program code. When the program product runs on a processor of an electronic device, the program code is used to enable the processor of the electronic device to execute the steps of the method according to various exemplary implementations of the present application described above in this specification. For example, the electronic device can execute the emotion recognition model training method and / or emotion recognition method recorded in the embodiments of the present application.

[0094] The program product may employ any combination of one or more readable media. The readable medium may be a readable data medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0095] The implementation method of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. At least one processor can read the computer program from the computer-readable storage medium. When at least one processor executes the computer program, it can implement the emotion recognition model training method in the above-mentioned embodiment, and / or the technical solution of the emotion recognition method.

[0096] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices and computer program products according to the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable information processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable information processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0097] These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable information processing device to work in a specific manner, so that the instructions stored in the computer-readable memory produce a manufactured product including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0098] These computer program instructions can also be loaded onto a computer or other programmable information processing device so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0099] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary technical means in the art that are not disclosed in the present application.

[0100] It should be noted that, in addition to the implementation of the present application described in the above-mentioned specific embodiments, those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application is introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to the implementation. On the contrary, the purpose of introducing the invention in conjunction with the implementation is to cover other options or modifications that may extend based on the claims of the present application. In order to provide an in-depth understanding of the present application, the above description contains many specific details, and the present application may also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that, in the absence of conflict, the embodiments in the present application and the features in the embodiments can be combined with each other.

[0101] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.

[0102] Although the present application has been illustrated and described by referring to certain preferred implementations of the present application, it should be understood by those skilled in the art that the above contents are further detailed descriptions of the present application in combination with specific implementations, and it cannot be determined that the specific implementation of the present application is limited to these descriptions. Those skilled in the art may make various changes in form and details, including making several simple deductions or substitutions, without departing from the spirit and scope of the present application.

Claims

1. A method for training an emotion recognition model, characterized in that: The method comprises: Determine a training data set, the training data set comprising a plurality of emotion data samples, the plurality of emotion data samples being obtained based on image emotion information, sound emotion information, and text emotion information, and the plurality of emotion data samples comprising a plurality of unimodal emotion data samples and a plurality of multimodal emotion data samples, each of the unimodal emotion data samples comprising one of the image emotion information, the sound emotion information, and the text emotion information, and each of the multimodal emotion data samples comprising at least two of the image emotion information, the sound emotion information, and the text emotion information; The initial emotion recognition model is trained according to the training data set to obtain the emotion recognition model.

2. The emotion recognition model training method according to claim 1, characterized in that: The emotion data sample is obtained in the following way: Acquiring emotional information, the emotional information including the image emotional information, the sound emotional information and the text emotional information; Performing data enhancement processing on the emotion information to obtain the emotion data sample.

3. The emotion recognition model training method according to claim 2, characterized in that: Obtain emotional information, including: Get emotional video information; The emotional video information is subjected to information extraction processing to obtain the emotional information.

4. The emotion recognition model training method according to claim 3, characterized in that: Performing data enhancement processing on the emotion information to obtain the emotion data sample includes: Performing data enhancement processing on the image emotion information to obtain image data; Performing data enhancement processing on the sound emotion information to obtain sound data; Performing data enhancement processing on the text sentiment information to obtain text data; Performing data enhancement processing on the image emotion information and the sound emotion information to obtain image and sound data; Performing data enhancement processing on the image emotion information and the text emotion information to obtain image text data; Performing data enhancement processing on the voice emotion information and the text emotion information to obtain voice text data; Performing data enhancement processing on the image emotion information, the sound emotion information and the text emotion information to obtain image, sound and text data; The emotion data sample is obtained according to the image data, the sound data, the text data, the image and sound data, the image and text data, the sound and text data, and the image and sound and text data.

5. The emotion recognition model training method according to claim 3 or 4, characterized in that: The emotional video information is subjected to information extraction processing to obtain the emotional information, including: Performing image extraction processing on the emotional video information by using an image extraction algorithm to obtain the image emotional information; and Performing sound extraction processing on the emotional video information by using a sound extraction algorithm to obtain the sound emotional information; and The sound emotion information is subjected to text extraction processing by a text recognition algorithm to obtain the text emotion information.

6. The emotion recognition model training method according to any one of claims 1 to 5, characterized in that: Performing model training on the initial emotion recognition model according to the training data set to obtain the emotion recognition model includes: The emotion data samples are subjected to feature extraction processing to obtain emotion feature data, and the initial emotion recognition model is subjected to model training according to the emotion feature data to obtain the emotion recognition model.

7. The emotion recognition model training method according to claim 6, characterized in that: Performing feature extraction processing on the emotion data sample to obtain emotion feature data includes: Performing facial recognition processing on the emotion data sample including the image emotion information based on a facial recognition model to obtain a facial image, and performing feature extraction processing on the facial image based on a visual feature extractor to obtain image feature data; Performing sound feature extraction processing on the emotion data sample including the sound emotion information based on the sound feature extractor to obtain sound feature data; Performing text feature extraction processing on the emotion data sample including the text emotion information based on a text feature extractor to obtain text feature data; The image feature data, the sound feature data and the text feature data are used as the emotion feature data.

8. An emotion recognition method, characterized in that: The method comprises: Determine the user's emotional data, perform feature extraction processing on the emotional data to obtain emotional feature data, input the emotional feature data into an emotion recognition model to obtain an emotion recognition result of the user, and the emotion recognition model is obtained by model training based on the emotion recognition model training method described in any one of claims 1-7.

9. The emotion recognition method according to claim 8, characterized in that: Inputting the emotion feature data into an emotion recognition model to obtain an emotion recognition result of the user includes: The emotion feature data is input into the classifier of the emotion recognition model, so that the classifier determines the emotion category according to the emotion feature data as the emotion recognition result.

10. An electronic device, characterized in that: include: A processor, and a memory communicatively connected to the processor, wherein the processor includes a central processing unit and a graphics processing unit; The memory stores a computer program, and the processor executes the computer program stored in the memory so that the electronic device implements the emotion recognition model training method as described in any one of claims 1 to 7, and / or the emotion recognition method as described in claim 8 or 9.

Images