Emotion determination model training method and device and emotion determination method and device

Abstract

The invention relates to the technical field of machine learning, in particular to an emotion determination model training method, an emotion determination model training device, an emotion determination method and device, a computer readable medium and electronic equipment. The training data comprises multiple pieces of user sample data and emotion tags corresponding to the user sample data, and the user sample data comprises at least one of face image data, text data and voice data; performing feature extraction on the user sample data to obtain a feature matrix corresponding to each user sample data; and training the initial emotion determination model according to the feature matrix corresponding to the user sample data and the emotion label corresponding to the user sample data to obtain a target emotion determination model. The technical scheme of the embodiment of the invention overcomes the technical problem of overlarge model memory occupation.

Description

Background Art

[0002] People's daily activities or behaviors are often affected by emotions. The quality of emotions affects our mood, work, life, and even life. Therefore, it is of great practical significance to identify the emotions of end-side users, track people's psychological states, and recommend some content or suggestions that can help alleviate negative emotions when users have negative emotions.

[0003] However, the emotion determination method in the prior art needs to use different models for different input data, which is difficult to operate. When the input data includes multiple types, multiple different types of input heads are required, and the model memory occupies too much. Summary of the invention

[0004] The purpose of the present disclosure is to provide a training method for an emotion determination model, a training device for an emotion determination model, an emotion determination method and device, a computer-readable medium and an electronic device, thereby overcoming the technical problem of excessive model memory usage at least to a certain extent.

[0005] According to a first aspect of the present disclosure, a method for training an emotion determination model is provided, comprising: acquiring training data, the training data comprising a plurality of user sample data and an emotion label corresponding to each user sample data, wherein the user sample data comprises at least one of face image data, text data and voice data; performing feature extraction on the user sample data to obtain a feature matrix corresponding to each user sample data; and training an initial emotion determination model according to the feature matrix corresponding to the user sample data and the emotion label corresponding to the user sample data to obtain a target emotion determination model.

[0006] According to a second aspect of the present disclosure, there is provided an emotion determination method, comprising: obtaining input data, the input data comprising at least one of facial image data, text data and voice data; performing feature extraction on the input data to obtain a target feature matrix corresponding to each input data; inputting the target feature matrix into a target emotion determination model to obtain a target emotion corresponding to the input data; wherein the target emotion determination model is trained by a training method for an emotion determination model.

[0007] According to a third aspect of the present disclosure, a training device for an emotion determination model is provided, comprising: a data acquisition module, used to acquire training data, the training data comprising a plurality of user sample data and an emotion label corresponding to each user sample data, wherein the user sample data comprises at least one of face image data, text data and voice data; a feature extraction module, used to perform feature extraction on the user sample data to obtain a feature matrix corresponding to each user sample data; and a model training module, used to train an initial emotion determination model according to the feature matrix corresponding to the user sample data and the emotion label corresponding to the user sample data to obtain a target emotion determination model.

[0008] According to a fourth aspect of the present disclosure, an emotion determination device is provided, comprising: a data input module, used to obtain input data, the input data including at least one of facial image data, text data and voice data; a feature acquisition module, used to perform feature extraction on the input data to obtain a target feature matrix corresponding to each input data; an emotion determination module, used to input the target feature matrix into a target emotion determination model to obtain a target emotion corresponding to the input data; wherein the target emotion determination model is trained by a training method of an emotion determination model.

[0009] According to a fifth aspect of the present disclosure, a computer-readable medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the above method is implemented.

[0010] According to a sixth aspect of the present disclosure, an electronic device is provided, characterized in that it includes: one or more processors; and a memory for storing one or more programs, which enables the one or more processors to implement the above method when the one or more programs are executed by the one or more processors.

[0011] An embodiment of the present disclosure provides a training method for an emotion determination model. On the one hand, when training the initial emotion determination model, the user sample data used includes at least one of facial image data, text data and voice data, thereby enriching the training data and making the obtained target emotion determination model more accurate. On the other hand, before training, at least one of the facial image data, text data and voice data is subjected to feature extraction to obtain feature matrices of the same type, thereby ensuring the uniformity of the input data, reducing the difficulty of model design, eliminating the need to set different input interfaces to meet different input data, reducing the memory usage of the model, facilitating the layout of the target emotion determination model on a mobile terminal with smaller memory, and increasing the application breadth of the emotion determination model.

[0012] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the specification are used to explain the principles of the present disclosure. Obviously, the accompanying drawings described below are only some embodiments of the present disclosure, and for ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without creative work.

[0014] Figure 1 A schematic diagram showing an exemplary system architecture to which the embodiments of the present disclosure can be applied is shown.

[0015] Figure 2 A flowchart schematically illustrates a method for training an emotion determination model in an exemplary embodiment of the present disclosure.

[0016] Figure 3 A schematic diagram of a text rendering grayscale image in an exemplary embodiment of the present disclosure is shown.

[0017] Figure 4 A schematic diagram showing a method for acquiring a feature matrix in an exemplary embodiment of the present disclosure.

[0018] Figure 5 A schematic diagram showing a grayscale image in an exemplary embodiment of the present disclosure.

[0019] Figure 6 A schematic diagram showing a feature matrix corresponding to facial image data in an exemplary embodiment of the present disclosure.

[0020] Figure 7 The flowchart of a method for determining an emotion in an exemplary embodiment of the present disclosure is schematically shown.

[0021] Figure 8 A data flow diagram of a target emotion determination model in an exemplary embodiment of the present disclosure is schematically shown.

[0022] Fig. 9 A structural diagram of a transformer encoder in an exemplary embodiment of the present disclosure is shown.

[0023] Fig.10 A schematic diagram showing a general classification layer in an exemplary embodiment of the present disclosure.

[0024] Fig.11 A schematic diagram showing another classification layer in an exemplary embodiment of the present disclosure.

[0025] Fig.12 A flow chart showing another emotion determination method in an exemplary embodiment of the present disclosure.

[0026] Fig.13 The flowchart of the vector matching search in the exemplary embodiment of the present disclosure is schematically shown.

[0027] Fig.14 A schematic diagram showing a vector matching search result in an exemplary embodiment of the present disclosure.

[0028] Fig.15 The figure schematically shows the composition of a training device for an emotion determination model in an exemplary embodiment of the present disclosure.

[0029] Fig.16 The figure schematically shows the composition of the emotion determination device in the exemplary embodiment of the present disclosure.

[0030] Fig.17 A schematic diagram of an electronic device to which the embodiments of the present disclosure can be applied is shown. DETAILED DESCRIPTION

[0031] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the disclosure will be more comprehensive and complete and to fully convey the concepts of the example embodiments to those skilled in the art. The described features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0032] In addition, the accompanying drawings are only schematic illustrations of the present disclosure and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0033] People's daily activities or behaviors are often affected by emotions. Positive emotions can make people feel happy and relaxed, and they are more optimistic about doing things. Negative emotions can make people restless and tense, which can inhibit their ability to move. Long-term negative emotions can make the body sick. The quality of emotions affects our mood, work, life, and even life. Therefore, it is of great practical significance to identify the emotions of end-side users, track their psychological state, and recommend some content or suggestions that can help alleviate negative emotions when users have negative emotions.

[0034] The user's emotions can often be identified from multimodal information such as facial expressions in video chats or selfies, voice content and speaking tone in voice chats, and text content when sending text messages. Machine learning models can gain insight into a person's emotional fluctuations through these data. In other words, the current input data is generally multiple different types of input data. However, the emotion determination method in the prior art needs to use different models for different input data, which is difficult to operate. When the input data includes multiple types, multiple different types of input heads are required, where the input head can be understood as an input interface. The input data is input into the above model through the input interface. Each input interface will increase the memory usage of the model, and multiple different types of input interfaces will make the model memory usage too large.

[0035] Based on the above shortcomings, the present disclosure provides a training method for an emotion determination model. Figure 1 A schematic diagram of a system architecture that can implement the training method of the above-mentioned emotion determination model is shown, and the system architecture 100 may include a terminal 110 and a server 120. The terminal 110 may be a terminal device such as a smart phone, a tablet computer, a desktop computer, a laptop computer, etc. The server 120 generally refers to a background system that provides emotion determination related services in this exemplary embodiment, and may be a server or a cluster formed by multiple servers. The terminal 110 and the server 120 may be connected via a wired or wireless communication link to perform data exchange.

[0036] In one embodiment, the training method of the above-mentioned emotion determination model can be executed by the terminal 110. For example, a user uses the terminal 110 to obtain training data, and the training data includes multiple user sample data and emotion labels corresponding to each user sample data, wherein the user sample data includes at least one of face image data, text data and voice data, and the terminal 110 extracts features from the image of the user sample data to obtain a feature matrix corresponding to each user sample data; the initial emotion determination model is trained according to the feature matrix corresponding to the user sample data and the emotion labels corresponding to the user sample data to obtain a target emotion determination model, and the target emotion determination model is output.

[0037] In one embodiment, the training method of the above-mentioned emotion determination model can be executed by the server 120. For example, a user uses the terminal 110 to obtain training data, and the training data includes multiple user sample data and emotion labels corresponding to each user sample data, wherein the user sample data includes at least one of face image data, text data and voice data. After the terminal 110 obtains the training data, the terminal 110 uploads the training data to the server 120, and the server 120 extracts features from the user sample data to obtain a feature matrix corresponding to each user sample data; the initial emotion determination model is trained according to the feature matrix corresponding to the user sample data and the emotion labels corresponding to the user sample data to obtain a target emotion determination model, and the target emotion determination model is returned to the terminal 110.

[0038] As can be seen from the above, the execution subject of the training method of the emotion determination model in this exemplary embodiment can be the above-mentioned terminal 110 or server 120, and the present disclosure does not limit this.

[0039] Combine the following Figure 2 The training method of the emotion determination model in this exemplary embodiment is described. Figure 2 An exemplary process of the training method of the emotion determination model is shown, which may specifically include steps S210 to S230.

[0040] refer to Figure 2 In step S210, training data is obtained, the training data including a plurality of user sample data and an emotion label corresponding to each user sample data, wherein the user sample data includes at least one of face image data, text data and voice data.

[0041] In an exemplary embodiment, training data may be first acquired, wherein the training data may include user sample data and emotion labels corresponding to the user sample data, wherein the user sample data may include at least one of facial image data, text data, and voice data, and the emotion labels may include emotions such as happiness, excitement, smile, normal, fear, anger, etc., and may also be customized according to user needs, which is not specifically limited in this exemplary implementation.

[0042] Optionally, a user sample data may include only one of facial image data, text data and voice data, or may include multiple facial image data, text data and voice data at the same time. For example, it may include facial image data and text data, or facial image data and voice data, or text data and voice data, that is, it may be any two of facial image data, text data and voice data, or may include facial image data, text data and voice data at the same time.

[0043] Preferably, during training, a user sample data includes facial image data, text data and voice data at the same time, so that the sample data is richer and the emotion determination model obtained through training has higher accuracy.

[0044] In an example implementation, during training, the training data may be divided into multiple sub-data, each of which may include data of one or more modalities, and each sub-data may simultaneously include data of three modalities: vision, speech, and text. The sample quantity distribution of the three modalities in the same sub-data may be determined according to actual conditions. Preferably, the proportion of the data of the three modalities in the same sub-data is 1 / 3 respectively.

[0045] In step S220, feature extraction is performed on the user sample data to obtain a feature matrix corresponding to each user sample data.

[0046] After obtaining the above-mentioned user sample data, the processor can perform feature extraction on the above-mentioned user sample data to obtain a feature matrix corresponding to the user sample data, wherein the above-mentioned feature matrix can be a three-channel feature matrix, and the three channels are grayscale image, position encoding matrix and modal embedding matrix.

[0047] Specifically, extracting features from user sample data to obtain a feature matrix corresponding to each user sample data may include steps S310 to S320.

[0048] In step S310, intermediate features are obtained based on user sample data.

[0049] In this example implementation, the grayscale image and modal information corresponding to the user sample data may be first obtained, and a modal embedding matrix corresponding to the sample data may be determined based on the modal information.

[0050] In an example implementation, when the user sample data includes facial image data, a facial grayscale image in the facial image data may be first acquired. The specific method for acquiring facial image data may refer to related technologies and will not be described in detail here.

[0051] When the above-mentioned user sample data is face image data, the modal information corresponding to the above-mentioned user sample data is image modality. At this time, the modal embedding matrix corresponding to the image modality can be queried in the repository, wherein the above-mentioned repository is located in the modal embedding module of the initial emotion determination model. When the above-mentioned initial emotion model is trained, the modal embedding matrix in the above-mentioned modal embedding module will also be updated.

[0052] Among them, the grayscale image of the above face image can be expressed as A n×n , the first modality embedding matrix corresponding to the above image modality can be expressed as

[0053] In another example implementation, the user sample data may be voice data. In this example implementation, at this time, the Mel-spectrogram grayscale image corresponding to the voice data may be first obtained. Specifically, the voice data may be first sampled at a preset sampling frequency to obtain a plurality of voice feature points, wherein the preset sampling frequency may be 16K, or a value greater than 16K, and may also be customized according to user needs, which is not specifically limited in this example implementation.

[0054] When sampling, the two-channel audio can be converted into mono audio and the audio waveform data can be normalized to [-1.0, 1.0] to obtain normalized waveform data, and then the preset size of the Mel spectrum grayscale image to be obtained can be determined. After obtaining the preset size, the above-mentioned speech data is framed according to the above-mentioned preset size to obtain multiple waveform data segments composed of speech feature points. For example, as mentioned above, assuming that the duration of each frame is 25ms, the number of data points contained in each frame is 0.025f, where f is the preset sampling frequency of the audio.

[0055] Then, each frame of waveform data fragment can be windowed, and the windowed waveform data can be fast Fourier transformed based on the preset size to obtain spectrum data of a preset size. Specifically, each frame of waveform data is windowed, wherein the window can be a Hanning window, a Hamming window, a Blackman window, etc., and can also be customized according to user needs. No specific limitation is made in this example implementation. Windowing can alleviate the problem of spectrum leakage.

[0056] Then, an N-point fast Fourier transform FFT or STFT is used, where N is an exponent of 2, which can be 512, 1024, 2048, etc., to obtain the spectrum data of each frame, which is then converted into a Mel frame through Mel filtering and the logarithm log is taken.

[0057] It should be noted that when the above preset size is n×n, when extracting the Mel spectrum, each frame of speech waveform data can be converted into a Mel frame vector with a dimension of n according to the above preset size and the logarithm log is taken. At this time, the duration of each frame can be determined according to the total length of the speech data and the value of n. When framing the audio waveform data, select a reasonable frame length (that is, the duration of each frame of audio). After obtaining the audio frame data, intercept the first n frames and extract Mel frames for these n frames. When extracting the Mel frame, set the number of mel_bins to n, then n Mel frame vectors with a dimension of n can be obtained, which are stacked into a matrix of size n×n, namely B. m×m , that is, Mel grayscale image.

[0058] After obtaining the above mel grayscale image, the second modality embedding matrix corresponding to the above speech data can be obtained Specifically, the modal information corresponding to the speech data is the speech modality, and the modal embedding matrix corresponding to the speech modality can be queried in the repository. Among them, the above-mentioned storage library is located in the modal embedding module of the initial emotion determination model. When the above-mentioned initial emotion model is trained, the modal embedding matrix in the above-mentioned modal embedding module will also be updated.

[0059] In another example implementation, the user sample data may include text data. When obtaining a text rendering grayscale image corresponding to the text data, the preset size of the text rendering grayscale image to be obtained may be first determined, and then the number of texts in the text data may be determined. The size of each character in the text rendering image may be calculated based on the number of texts and the preset storage. For example, the preset size of the text rendering grayscale image is an m×m matrix. The pixel size of each character may be determined based on the number of texts. After obtaining the pixel size of each character, the text data may be rendered into a text rendering grayscale image of a preset size based on the character size, i.e., the pixel size.

[0060] In another example implementation, when obtaining a text rendering grayscale image corresponding to text data, you can first determine the preset size of the text rendering image and the size of each character (i.e., font size), and then calculate how many characters can be placed in each line based on the size of the rendering image and the font size of the characters. If the number of characters in the above text data is large and cannot be fully displayed in the text rendering image of the preset size, then the above text can be truncated.

[0061] Specifically, refer to Figure 3 As shown, the input text sequence can be rendered into an image in a "Z-shaped" manner. For example, the resolution of the preset feature map is 256*256, and the font size of each character is 32 pixels when the text is rendered. Then 8 lines of text can be arranged, and 8 text characters will be arranged in each line. The rendered image contains a maximum of 64 text characters.

[0062] In this example implementation, in actual situations, it is necessary to determine the size of the input image and the font size of the characters required by the model. m1 represents the maximum number of lines of text that can be arranged on the image, and m2 represents the maximum number of characters that can be arranged in each line of the image.

[0063]

[0064]

[0065] It should be noted that non-Chinese characters can be converted from half-width to full-width characters to ensure that each character is in the center of the text block in which it is located. Since the encoding module of the entire network architecture adopts a convolution-based approach and the convolution operation is local, rendering the input text sequence into an image in a "Z-shaped" manner can ensure that when the convolution kernel slides to the rightmost side of the text rendering grayscale image, the semantics of the text has local continuity, further improving the accuracy of the trained emotion determination model.

[0066] The text data is rendered into an n×n single-channel text rendering grayscale image C in the above manner. n×n After that, we can get the third modality embedding matrix corresponding to the above text modality: Specifically, the modal embedding matrix corresponding to the text modality can be queried in the repository, wherein the above-mentioned repository is located in the modal embedding module of the initial emotion determination model. When the above-mentioned initial emotion model is trained, the modal embedding matrix in the above-mentioned modal embedding module will also be updated.

[0067] In step S320, a position encoding matrix is ​​obtained, and a feature matrix is ​​determined according to the position encoding matrix and the intermediate features.

[0068] After obtaining the grayscale image and the corresponding modal embedding matrix corresponding to the sample data used by the user, the position coding matrix can be obtained, and then the position coding matrix, the grayscale image and the modal embedding matrix are stacked into the feature matrix.

[0069] Specifically, refer to Figure 4 As shown, when the user sample data is speech data, the Mel spectrum grayscale image obtained above can be stacked with the position coding matrix and the second modality embedding matrix to form a feature matrix. When the user sample data includes face image data, the face grayscale image, the position coding matrix and the first modality embedding matrix can be stacked to form a feature matrix. When the user sample data includes the text data, the text rendering grayscale image, the position coding matrix and the third modality embedding matrix can be stacked to form a three-channel feature matrix.

[0070] The specific process of obtaining the above position encoding matrix is ​​introduced in detail below.

[0071] In an exemplary embodiment of the present disclosure, referring to Figure 5 As shown, the feature matrix of each of the above modalities can be divided into sub-images. The speech modality in 5(a) is the result of converting the speech waveform data into an n×n single-channel image and then performing sub-image division; Figure 5 In (b), the face image is directly converted into an n×n single-channel grayscale image, and then the sub-image is divided; Figure 5 (c) The text data is dyed into an n×n single-channel image in a “Z-shaped” manner, and then the sub-image is divided.

[0072] The position coding matrix in the feature matrix here is an initialized position coding matrix. The specific values ​​of the initialized position coding matrix can be customized according to user needs and are not specifically limited in this example implementation.

[0073] It should be noted that the position coding matrix used in this example implementation is a static position coding matrix, and a dynamic position coding matrix, a relative position coding matrix, a rotation position coding matrix, etc. may also be used, which is not specifically limited in this example implementation.

[0074] In this example implementation, the position coding matrix may be divided to obtain a plurality of submodules of the position coding matrix, and the submodules may be numbered. For a specific numbering method, refer to Figure 5 As shown, the numbering can be performed in a "Z-shaped" manner.

[0075] Refer to the following Figure 6 , the calculation of the above position encoding matrix is ​​explained in detail, assuming Figure 6 The first channel is the grayscale image after the face image is converted, the second channel is the calculated position encoding matrix, and the third channel is the modal embedding matrix obtained by querying the modal embedding layer. Each channel is divided into several submodules. Assuming that the matrix size of each channel is n×n and the size of the submodule is k×k; then each channel will be divided into The value of each submodule of the second channel (i.e., the position encoding matrix) is calculated as follows:

[0076]

[0077]

[0078] Among them, pos represents the number of the current submodule, and Figure 6 Corresponding to the numbers in , k×k represents the size of the submodule, that is, a submodule contains k×k element values, 2i and 2i+1 represent the elements at the odd and even positions of the current submodule respectively, and the value of i ranges from 0 to (k×k) / 2.

[0079] The obtained position encoding matrix can be expressed as P n×n , then the feature matrix corresponding to the above face image data can be expressed as The feature matrix corresponding to the speech data can be expressed as The feature matrix corresponding to the text data can be expressed as

[0080] It should be noted that the calculation method of the above position coding matrix has nothing to do with the image data of the first channel and the third channel, that is, the first channel can also be a text rendering grayscale image or a Mel spectrum grayscale image, and the modal embedding matrix of the third channel is obtained based on the data query of the first channel. The calculation of the position coding matrix of the second channel is only related to the size of the submodule. Therefore, when the first channel is a text rendering grayscale image or a Mel spectrum grayscale image, the corresponding position coding matrix is ​​exactly the same as the position coding matrix corresponding to the first channel being a face image grayscale image.

[0081] The present invention converts text data, voice data, and face image data into a three-channel feature matrix for training an initial emotion determination model. The obtained target emotion model only needs to set up one input head to complete the emotion determination of data in the three modes of text, voice, and face image.

[0082] In step S230, the initial emotion determination model is trained according to the feature matrix corresponding to the user sample data and the emotion label corresponding to the user sample data to obtain a target emotion determination model.

[0083] After obtaining the feature matrix corresponding to the above-mentioned user sample data, the above-mentioned feature matrix can be input into the above-mentioned initial emotion determination module to obtain the reference emotion corresponding to the feature matrix, and then the loss function between the above-mentioned reference emotion and the above-mentioned emotion label is calculated, and the above-mentioned initial emotion determination model is trained based on the above-mentioned loss function until the above-mentioned initial emotion determination model meets the preset conditions to obtain the above-mentioned target emotion determination model.

[0084] To sum up, in this exemplary embodiment, on the one hand, when training the initial emotion determination model, the user sample data used includes at least one of facial image data, text data and voice data, which enriches the training data and makes the obtained target emotion determination model more accurate. On the other hand, before training, at least one of the facial image data, text data and voice data is feature extracted to obtain feature matrices of the same type, thereby ensuring the uniformity of the input data, reducing the difficulty of model design, eliminating the need to set different input interfaces to meet different input data, reducing the memory usage of the model, and facilitating the layout of the target emotion determination model on mobile terminals with smaller memory, thereby increasing the application breadth of the emotion determination model.

[0085] Further, see Figure 7 As shown, the present disclosure also provides an emotion determination method, which may include steps S710 to S730.

[0086] In step S710, input data is acquired, where the input data includes at least one of face image data, text data, and voice data.

[0087] In this example implementation, input data may be first acquired, and the input data may include at least one of face image data, text data, and the voice data.

[0088] Optionally, the input data may include only one of facial image data, text data and voice data, or may include multiple types of facial image data, text data and voice data at the same time.

[0089] In step S720, feature extraction is performed on the input data to obtain a target feature matrix corresponding to each input data.

[0090] After obtaining the above-mentioned input data, feature extraction can be performed on the above-mentioned input data to obtain a target feature matrix corresponding to each of the above-mentioned input data, wherein the process of obtaining the above-mentioned target feature matrix can refer to the determination method of the above-mentioned emotion determination model, and is not specifically limited in this example implementation.

[0091] In step S730, the target feature matrix is ​​input into the target emotion determination model to obtain the target emotion corresponding to the input data.

[0092] In this example implementation, the target emotion determination module may include a transformer encoder and a classification layer. After obtaining the target feature matrix, the target feature matrix may be firstly divided into modules.

[0093] The above target feature matrix is ​​a three-channel feature map, refer to Figure 8 As shown, the size of each channel can be n×n, and it can be divided into multiple sub-matrices, wherein each sub-matrix includes three-channel features, and its size can be k×k. The above sub-matrices can be stretched or reshaped into a vector, and the dimension of the vector can be 3×k×k.

[0094] Then, the vector sequence obtained by stretching each submatrix can be linearly transformed to obtain a transformed vector sequence. Specifically, the vector sequence obtained by stretching each submatrix can be input into the linear mapping layer for linear transformation to obtain the vector sequence corresponding to each submatrix, and then the above vector sequence can be input into the above transformer encoder to obtain the target feature vector.

[0095] Specifically, the above vector sequence is input into the transformer encoder for sequence modeling, and the first or last vector of the vector sequence weighted by the attention mechanism is selected as the target feature vector corresponding to the entire input data.

[0096] Reference Fig. 9The above-mentioned transformer encoder may include a first NORM layer, a multi-head self-attention mechanism layer, a second NORM layer and a perception layer. The specific functions of each layer in the transformer encoder can refer to the relevant technology and are not specifically limited in this example implementation.

[0097] After obtaining the target feature vector, the target feature vector may be input into the classification layer to obtain the corresponding target emotion.

[0098] In one example implementation, referring to Fig.10 As shown, the above classification layer can be a general classification type, and the above target feature vector can be input into the above general classification layer to obtain the target emotion, wherein the emotions in the general classification layer can include anger, disgust, fear, normal, sadness, surprise and happiness, etc., and can also be customized according to user needs, which is not specifically limited in the real-time verification of this example.

[0099] In this example implementation, multiple modal outputs use the same set of emotion labels, that is, the emotion labels of multiple modalities are aligned to their states, which can adapt to the situation when the labels of training data of different modalities are aligned. The output results do not need to be judged again, which improves the convenience of using the target emotion model.

[0100] In another example implementation, referring to Fig.11 As shown, the above-mentioned classification layer may include a speech classification layer, a text classification layer and an image classification layer. At this time, the modal information of the above-mentioned target feature vector may be determined first, and the corresponding target classification layer may be determined according to the above-mentioned modal information. Then, the above-mentioned target feature vector may be input into the above-mentioned target classification layer to obtain the target emotion.

[0101] For example, the emotions corresponding to the audio classification layer may include anger, disgust, fear, normal, sadness, surprise, excitement and happiness, etc., the emotions corresponding to the image classification layer may include anger, disgust, fear, normal, sadness, surprise, happiness, smiling and crying, etc., the emotions corresponding to the text classification layer may include anger, disgust, fear, normal, sadness, surprise and happiness, etc., and may also be customized according to user needs, and are not specifically limited in the real-time verification of this example.

[0102] In this example implementation, if the input data is input data within a continuous time region and includes data of multiple modalities, then all candidate emotions appearing within the time region can be obtained, and the candidate emotion that appears most frequently here can be used as the target emotion.

[0103] If the time period in the above input data is short, if the target emotion corresponding to the above facial image data is not neutral, the target emotion corresponding to the above facial image data is prioritized; if the target emotion corresponding to the facial image data is neutral, but the target emotion corresponding to the voice data is not neutral, the target emotion corresponding to the voice data is prioritized; if the target emotion corresponding to the above voice data is neutral, the target emotion corresponding to the text data is prioritized. The specific determination method can also be customized according to user needs, and is not specifically limited in this example implementation.

[0104] In this example implementation, different emotion labels may be output for different modalities, and specific target emotions may be determined according to different scenarios. The obtained target emotions may refer to scenario factors, thereby improving the output accuracy of the target emotion determination model.

[0105] Refer to the following Fig.12 The above emotion determination model is described in detail. First, data collection can be performed. Specifically, step 1210 can be executed to determine whether voice interaction is performed; step S1220 can be executed to determine whether a selfie photo exists; and step S1230 can be executed to determine whether text input is performed; specifically, it can be determined whether the user is performing voice interaction or saving a recording. If so, the audio segment to be identified is intercepted and cached in the data cache device on the end side, and it is determined whether the user has saved a face photo just taken with the front camera in the album or whether the user is performing video interaction. If so, the corresponding face picture is cached in the data cache device on the end side, and then it is determined whether the user is using the input method to input text data. If so, the corresponding text sequence is cached in the data cache device on the end side.

[0106] Then, step S1240 can be executed for data collection. Specifically, input data is obtained from the data storage device, and then step S1250 is executed to determine whether the data is empty. If not, step S1260 is executed for feature extraction. Specifically, if no data is collected in the data cache device on the end side, data collection is performed again after waiting for the preset next data collection time or the next data collection trigger event. If there is data in the data cache device on the end side, feature extraction is performed on the above input data to obtain the corresponding target feature matrix. Then step S1270 is executed to perform model prediction. Specifically, the target feature matrix of step S1260 is input into the above target emotion determination module to obtain the corresponding target emotion of the input data. If the collected data contains multiple samples or data of multiple modes, a comprehensive judgment of the emotion category can be made based on the results of multiple samples predicted by the model.

[0107] After obtaining the user's target emotion, intelligent decisions can be made on the mobile phone. For example, if the user is depressed, apps or interesting content that can help alleviate negative emotions can be recommended to the user. It is also possible to guide the user's emotions in the form of message reminders. After completing the above steps, clear the data cache device and wait for the preset next data collection time or the next data collection trigger event, and then execute the above steps for feature extraction and model prediction. Data collection will be carried out with the user's permission to use related services, and emotion recognition will only be performed on the end side, and the data will not be uploaded to the cloud. Since the model is deployed on the user's terminal device, data collection and model reasoning are completed on the end-side device, which effectively protects the user's personal privacy and data security.

[0108] In an exemplary embodiment of the present disclosure, the feature extraction module in the target emotion determination module can be used to extract features from text data, voice data, and corresponding face image data to obtain a corresponding target vector. For example, referring to Fig.13 As shown, the text data and the face image data can be feature extracted to obtain the target text vector and the target face image vector, and the face image corresponding to the text data can be selected from the face image database according to the similarity between the target text vector and the target face image vector. Fig.14 As shown, if the input text data is a picture of a happy mood, the corresponding happy mood image can be selected and displayed in the database using the above method. It should be noted that face image retrieval can also be performed through voice data. The specific retrieval method is the same as the method of retrieving face image data through text data, which will not be repeated here.

[0109] It should be noted that the above figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0110] For further reference, Fig.15 As shown, in the embodiment of this example, a training device 1500 for an emotion determination model is also provided, including a data acquisition module 1510, a feature extraction module 1520, and a model training module 1530. Among them:

[0111] The data acquisition module 1510 can be used to acquire training data, which includes multiple user sample data and emotion labels corresponding to each user sample data, wherein the user sample data includes at least one of facial image data, text data and voice data; the feature extraction module 1520 can be used to extract features from the user sample data to obtain a feature matrix corresponding to each user sample data; the model training module 1530 can be used to train the initial emotion determination model according to the feature matrix corresponding to the user sample data and the emotion labels corresponding to the user sample data to obtain a target emotion determination model.

[0112] In an example implementation, the feature extraction module 1520 may be configured to obtain intermediate features based on user sample data; obtain a position coding matrix, and determine a feature matrix based on the position coding matrix and the intermediate features.

[0113] When acquiring intermediate features based on user sample data, the feature extraction module 1520 can be configured to determine the corresponding grayscale image and modal information of the user sample data, and determine the modal embedding matrix corresponding to the user sample data based on the modal information. The specific user sample data includes face image data, and determining the corresponding grayscale image and modal information of the user sample data, and determining the modal embedding matrix corresponding to the user sample data based on the modal information includes: obtaining the face grayscale image in the face image data; obtaining the first modal embedding matrix corresponding to the face image data. The user sample data includes voice data, and determining the corresponding grayscale image and modal information of the user sample data, and determining the modal embedding matrix corresponding to the user sample data based on the modal information includes: obtaining the Mel spectrum grayscale image in the voice data; obtaining the second modal embedding matrix corresponding to the voice data.

[0114] In an example implementation, obtaining a Mel-spectrogram grayscale image in speech data includes: sampling the speech data at a preset sampling frequency to obtain a plurality of speech feature points; determining a preset size of the Mel-spectrogram grayscale image, and performing frame processing on the speech data according to the preset size to obtain a plurality of waveform data consisting of speech feature points; performing windowing processing on each waveform data, and obtaining spectrum data of a preset size based on performing a fast Fourier transform on the windowed waveform data; and performing Mel filtering on the spectrum data according to the preset size to obtain a Mel-spectrogram grayscale image.

[0115] In an example implementation, the user sample data includes text data, and the feature extraction module 1520 can be configured to obtain a text rendering grayscale image corresponding to the text data; obtain a third modality embedding matrix corresponding to the text data. Specifically, determine a preset size of the text rendering grayscale image; obtain the number of texts in the text data, determine the text size in the text rendering grayscale image according to the number of texts and the preset size; and render the text data into a text rendering grayscale image based on the text size.

[0116] In an example implementation, when obtaining the position coding matrix, the feature extraction module 1520 may be configured to obtain an initialized position coding matrix; divide the position coding matrix into a plurality of sub-modules, and number each sub-module; calculate the coding matrix corresponding to each sub-module according to the number of the sub-module; and determine the position coding matrix according to the coding matrix corresponding to each sub-module.

[0117] In an example implementation, the model training module 1530 may be configured to input the feature matrix into the initial emotion determination model to obtain a reference emotion corresponding to the feature matrix; update the initial emotion determination model based on the reference emotion and the emotion label until the initial emotion determination model meets a preset condition to obtain a target emotion determination model.

[0118] For further reference, Fig.16 As shown, in the embodiment of this example, an emotion determination device 1600 is also provided, including a data input module 1610, a feature acquisition module 1620, and an emotion determination module 1630. Among them:

[0119] The data input module 1610 can be used to obtain input data, which includes at least one of facial image data, text data and voice data; the feature acquisition module 1620 can be used to extract features from the input data to obtain a target feature matrix corresponding to each input data; the emotion determination module 1630 can be used to input the target feature matrix into a target emotion determination model to obtain a target emotion corresponding to the input data; wherein the target emotion determination model is trained by a training method of an emotion determination model.

[0120] In an example implementation of the present disclosure, the target emotion determination model includes a transformer encoder and a classification layer, and the emotion determination module 1630 can be configured to encode the target feature matrix based on the transformer encoder to obtain a target feature vector; and determine the target emotion corresponding to the target feature matrix using the classification layer based on the target feature vector. Specifically, the target feature matrix is ​​divided into a plurality of sub-matrices; each sub-matrix is ​​linearly changed to obtain a vector sequence corresponding to each sub-matrix; and each vector sequence is input into the transformer encoder to obtain the target feature vector.

[0121] In an exemplary embodiment of the present disclosure, the classification layer includes a general classification layer, and the emotion determination module 1630 may be configured to input the target feature vector into the general classification layer to obtain the target emotion.

[0122] In an example implementation of the present disclosure, the classification layer includes an audio classification layer, a text classification layer, and an image classification layer, and 1630 can be configured to obtain modal information in a target feature vector; determine a target classification layer corresponding to the target feature vector based on the modal information; and input the target feature vector into the target classification layer to obtain a target emotion.

[0123] The specific details of each module in the above device have been described in detail in the implementation method of the method part. The undisclosed details can be found in the implementation method of the method part, so they will not be repeated here.

[0124] The exemplary embodiment of the present disclosure also provides an electronic device for executing the training method of the above-mentioned emotion determination model, and the electronic device may be the above-mentioned terminal 110 or the server 120. Generally, the electronic device may include a processor and a memory, the memory is used to store executable instructions of the processor, and the processor is configured to execute the above-mentioned emotion determination model training method by executing the executable instructions.

[0125] Below Fig.17 The mobile terminal 1700 in FIG. 1 is taken as an example to exemplify the structure of the electronic device. It should be understood by those skilled in the art that, in addition to the components specifically used for mobile purposes, Fig.17 The construction in can also be applied to fixed type equipment.

[0126] like Fig.17 As shown, the mobile terminal 1700 may specifically include: a processor 1701, a memory 1702, a bus 1703, a mobile communication module 1704, an antenna 1, a wireless communication module 1705, an antenna 2, a display screen 1706, a camera module 1707, an audio module 1708, a power module 1709 and a sensor module 1710.

[0127] Processor 1701 may include one or more processing units, for example: processor 1701 may include AP (Application Processor), modem processor, GPU (Graphics Processing Unit), ISP (Image Signal Processor), controller, encoder, decoder, DSP (Digital Signal Processor), baseband processor and / or NPU (Neural-Network Processing Unit), etc. The training method of the emotion determination model in this exemplary embodiment may be executed by AP, GPU or DSP, and when the method involves neural network-related processing, it may be executed by NPU.

[0128] The processor 1701 may be connected to the memory 1702 or other components via a bus 1703 .

[0129] The memory 1702 may be used to store computer executable program codes, which include instructions. The processor 1701 executes various functional applications and data processing of the mobile terminal 1700 by running the instructions stored in the memory 1702. The memory 1702 may also store application data, such as images, videos, and other files.

[0130] The communication function of the mobile terminal 1700 can be implemented through the mobile communication module 1704, antenna 1, wireless communication module 1705, antenna 2, modulation and demodulation processor and baseband processor. Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. The mobile communication module 1704 can provide 2G, 3G, 4G, 5G and other mobile communication solutions applied to the mobile terminal 1700. The wireless communication module 1705 can provide wireless communication solutions such as wireless LAN, Bluetooth, near field communication, etc. applied to the mobile terminal 1700.

[0131] The display screen 1706 is used to implement display functions, such as displaying user interfaces, images, and videos. The camera module 1707 is used to implement shooting functions, such as shooting images and videos. The audio module 1708 is used to implement audio functions, such as playing audio and collecting voice. The power module 1709 is used to implement power management functions, such as charging the battery, powering the device, and monitoring the battery status. The sensor module 1710 may include a depth sensor 17101, a pressure sensor 17102, a gyroscope sensor 17103, an air pressure sensor 17104, etc., to implement corresponding sensing detection functions.

[0132] Those skilled in the art will appreciate that various aspects of the present disclosure may be implemented as systems, methods or program products. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as "circuits", "modules" or "systems".

[0133] The exemplary embodiments of the present disclosure also provide a computer-readable storage medium on which a program product capable of implementing the above-mentioned method of the present specification is stored. In some possible implementations, various aspects of the present disclosure may also be implemented in the form of a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section of the present specification.

[0134] It should be noted that the computer-readable medium shown in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0135] In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, apparatus, or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in combination with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wireless, wire, optical cable, RF, etc., or any suitable combination of the above.

[0136] In addition, program code for performing the operations of the present disclosure may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0137] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and examples are to be considered as exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

[0138] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A method for training an emotion determination model, characterized in that: include: Acquire training data, wherein the training data includes a plurality of user sample data and an emotion label corresponding to each of the user sample data, wherein the user sample data includes at least one of face image data, text data, and voice data; Extracting features from the user sample data to obtain a feature matrix corresponding to each of the user sample data; The initial emotion determination model is trained according to the feature matrix corresponding to the user sample data and the emotion label corresponding to the user sample data to obtain a target emotion determination model.

2. The method according to claim 1, characterized in that The extracting features of the user sample data to obtain a feature matrix corresponding to each user sample data includes: Acquire intermediate features according to the user sample data; A position encoding matrix is ​​obtained, and the feature matrix is ​​determined according to the position encoding matrix and the intermediate features.

3. The method according to claim 2, characterized in that The intermediate features include a modality embedding matrix; The acquiring of intermediate features according to the user sample data comprises: Determine the grayscale image and modal information corresponding to the user sample data, and determine the modal embedding matrix corresponding to the user sample data based on the modal information.

4. The method according to claim 3, characterized in that Determining the feature matrix according to the position encoding matrix and the intermediate features comprises: The position encoding matrix, the grayscale image, and the modality embedding matrix are stacked into the feature matrix.

5. The method according to claim 3, characterized in that: The user sample data includes face image data, and determining the grayscale image and modal information corresponding to the user sample data, and determining the modal embedding matrix corresponding to the user sample data according to the modal information includes: Acquire a grayscale image of a face from the face image data; Obtain a first modality embedding matrix corresponding to the facial image data.

6. The method according to claim 3, characterized in that The user sample data includes voice data, and the determining of the grayscale image and modal information corresponding to the user sample data, and determining the modal embedding matrix corresponding to the user sample data according to the modal information includes: Obtaining a Mel-spectrogram grayscale image in the speech data; Obtain a second modality embedding matrix corresponding to the speech data.

7. The method according to claim 6, characterized in that The obtaining of the Mel frequency spectrum grayscale image in the speech data comprises: Sampling the speech data according to a preset sampling frequency to obtain a plurality of speech feature points; Determining a preset size of the Mel-spectrogram grayscale image, and performing frame processing on the speech data according to the preset size to obtain a plurality of waveform data consisting of speech feature points; Performing windowing processing on each of the waveform data, and obtaining spectrum data of a preset size based on performing fast Fourier transform on the windowed waveform data; The mel-spectrum grayscale image is obtained by performing mel filtering on the spectrum data according to the preset size.

8. The method according to claim 3, characterized in that The user sample data includes text data, and determining the grayscale image and modal information corresponding to the user sample data, and determining the modal embedding matrix corresponding to the user sample data according to the modal information includes: Obtain a text rendering grayscale image corresponding to the text data; Get the third modality embedding matrix corresponding to the text data.

9. The method according to claim 8, characterized in that The obtaining of the text rendering grayscale image corresponding to the text data comprises: Determine the preset size of the grayscale image for text rendering; Acquire the number of texts in the text data, and determine the text size in the text rendering grayscale image according to the number of texts and the preset size; The text data is rendered into the text rendering grayscale image based on the text size.

10. The method according to any one of claims 2, characterized in that The obtaining of the position coding matrix comprises: Get the initialization position encoding matrix; Dividing the position coding matrix into a plurality of submodules, and numbering each of the submodules; Calculate the encoding matrix corresponding to each submodule according to the number of the submodule; The position coding matrix is ​​determined according to the coding matrices corresponding to the submodules.

11. The method according to claim 1, characterized in that: Training the initial emotion determination model according to the feature matrix corresponding to the user sample data and the emotion label corresponding to the user sample data to obtain the target emotion determination model includes: Inputting the feature matrix into the initial emotion determination model to obtain a reference emotion corresponding to the feature matrix; The initial emotion determination model is updated based on the reference emotion and the emotion label until the initial emotion determination model meets a preset condition, so as to obtain the target emotion determination model.

12. A method for determining emotion, characterized in that: include: Acquiring input data, wherein the input data includes at least one of face image data, text data, and voice data; Performing feature extraction on the input data to obtain a target feature matrix corresponding to each of the input data; Inputting the target feature matrix into a target emotion determination model to obtain a target emotion corresponding to the input data; Wherein, the target emotion determination model is trained by the method described in any one of claims 1 to 11.

13. The method according to claim 12, characterized in that The target emotion determination model includes a transformer encoder and a classification layer; The step of inputting the target feature matrix into the target emotion determination model to obtain the target emotion corresponding to the input data includes: Encoding the target feature matrix based on the transformer encoder to obtain a target feature vector; The target emotion corresponding to the target feature matrix is ​​determined using the classification layer based on the target feature vector.

14. The method according to claim 13, characterized in that The step of encoding the target feature matrix based on the transformer encoder to obtain a target feature vector includes: Dividing the target feature matrix into a plurality of sub-matrices; Performing linear transformation on each of the sub-matrices to obtain a vector sequence corresponding to each of the sub-matrices; Each of the vector sequences is input into the transformer encoder to obtain a target feature vector.

15. The method according to claim 13, characterized in that The classification layer includes a general classification layer, and determining the target emotion corresponding to the target feature matrix based on the target feature vector using the classification layer includes: The target feature vector is input into the general classification layer to obtain the target emotion.

16. The method according to claim 13, characterized in that The classification layer includes an audio classification layer, a text classification layer and an image classification layer, and determining the target emotion corresponding to the target feature matrix based on the target feature vector by using the classification layer includes: Determining modal information in the target feature vector; Determine a target classification layer corresponding to a target feature vector according to the modal information; The target feature vector is input into the target classification layer to obtain the target emotion.

17. A training device for an emotion determination model, characterized in that: include: A data acquisition module, used to acquire training data, wherein the training data includes a plurality of user sample data and an emotion label corresponding to each of the user sample data, wherein the user sample data includes at least one of face image data, text data and voice data; A feature extraction module, used to extract features from the user sample data to obtain a feature matrix corresponding to each user sample data; The model training module is used to train the initial emotion determination model according to the feature matrix corresponding to the user sample data and the emotion label corresponding to the user sample data to obtain the target emotion determination model.

18. An emotion determination device, characterized in that: include: A data input module, used to obtain input data, wherein the input data includes at least one of face image data, text data and voice data; A feature acquisition module, used to extract features from the input data to obtain a target feature matrix corresponding to each input data; An emotion determination module, used for inputting the target feature matrix into a target emotion determination model to obtain a target emotion corresponding to the input data; Wherein, the target emotion determination model is trained by the method described in any one of claims 1 to 11.

19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 16 is implemented.

20. An electronic device, characterized in that: include: one or more processors; as well as A memory for storing one or more programs, which, when executed by the one or more processors, enables the one or more processors to implement the method according to any one of claims 1 to 16.

Images