Power personnel emotion recognition method, device, electronic equipment and medium

Abstract

The present application discloses a method, device, electronic device, and medium for emotion recognition of electric power personnel. The method may include: extracting speech unimodal features and expression unimodal features respectively; using speech unimodal features and expression unimodal features as two inputs of a Siamese network, calculating the similarity loss function of the two outputs, and extracting shared information of the two modalities; extracting specific information of speech unimodal features and expression unimodal features through orthogonal constraints to obtain an orthogonal loss function; performing emotion classification based on the shared information and specific information to obtain a total loss function; and obtaining the final emotion classification by minimizing the total loss function. The present invention effectively reduces the distribution gap and information redundancy between the two modalities caused by the complex and high-noise working environment of electric power personnel, thereby improving the performance of emotion recognition of electric power personnel.

Description

Technical Field

[0001] The present invention relates to the field of emotion recognition, and more specifically, to a method, device, electronic equipment and medium for emotion recognition of power personnel. Background Art

[0002] Power safety is a crucial safeguard for national security. The safe and reliable operation of infrastructure in sectors such as finance, communications, transportation, water supply, and gas supply all rely on a continuous and stable power supply. Furthermore, power safety is closely intertwined with political, economic, cyber, and social security. A large-scale power outage could trigger a cross-domain chain reaction, causing significant economic losses, social panic, and threatening national security. In the power industry, most safety incidents are caused by operator errors, specifically due to weak safety awareness, poor emergency response capabilities, and poor physical and emotional health. Generally speaking, the emotional state of power workers significantly impacts their mental state and physical coordination. For example, sadness or fear can easily lead to misjudgment and operational errors, resulting in safety incidents. Therefore, using artificial intelligence (AI) to proactively identify power workers' emotions can effectively reduce operational violations and errors, safeguarding national power safety. However, the complex and varied working environments of power workers, coupled with high levels of noise, result in significant distribution gaps and information redundancy in speech and facial expressions in power worker emotion datasets, making it difficult to effectively identify their emotional states.

[0003] Therefore, it is necessary to develop a method, device, electronic equipment and medium for emotion recognition of power personnel.

[0004] The information disclosed in the background technology section of the present invention is only intended to deepen the understanding of the general background technology of the present invention, and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the Invention

[0005] The present invention proposes a method, device, electronic device and medium for emotion recognition of power personnel, which can effectively reduce the distribution gap and information redundancy between the two modes caused by the complex and high-noise working environment of power personnel, and improve the performance of emotion recognition of power personnel.

[0006] In a first aspect, an embodiment of the present disclosure provides a method for identifying emotions of electric power personnel, comprising:

[0007] Extract speech unimodal features and expression unimodal features respectively;

[0008] The speech unimodal feature and the expression unimodal feature are respectively used as two inputs of the Siamese network, and a similarity loss function of the two outputs is calculated to extract the shared information of the two modalities;

[0009] Extracting specific information of the speech unimodal feature and the expression unimodal feature through orthogonal constraints to obtain an orthogonal loss function;

[0010] Perform sentiment classification based on the shared information and the unique information to obtain a total loss function;

[0011] The final sentiment classification is obtained by minimizing the total loss function.

[0012] Preferably, the similarity loss function is:

[0013]

[0014] in, is the similarity loss function, represents a set of emotion categories, and denote the shared features of speech and expression modalities, (·) i represents the features of the i-th sentiment category, and |·| and ||·|| represent the L1 and L2 norms, respectively.

[0015] Preferably, extracting specific information of the speech unimodal feature and the expression unimodal feature through orthogonal constraints to obtain an orthogonal loss function includes:

[0016] Extracting corresponding unique information according to the voice unimodal feature and the expression unimodal feature respectively;

[0017] The orthogonal loss function is calculated based on the shared information and the unique information of the two modalities.

[0018] Preferably, the orthogonal loss function is:

[0019]

[0020] in, is the orthogonal loss function, is a unique feature of speech, It is a characteristic of facial expression. Represents the dot product operation.

[0021] Preferably, obtaining the total loss function comprises:

[0022] Calculate the cross entropy loss function;

[0023] The total loss function is obtained according to the cross entropy loss function, the similarity loss function, and the orthogonal loss function.

[0024] Preferably, the cross entropy loss function is:

[0025]

[0026] in, is the cross entropy loss function, y i represents the sentiment label of the i-th sample, Represents the sentiment classification module C M (·) The predicted label for the i-th sample.

[0027] Preferably, the total loss function is:

[0028]

[0029] in, is the total loss function, and α represents a hyperparameter.

[0030] As a specific implementation of the embodiment of the present disclosure,

[0031] In a second aspect, the present disclosure also provides an emotion recognition device for electric personnel, including:

[0032] The unimodal feature extraction module extracts speech unimodal features and expression unimodal features respectively;

[0033] a feature extraction module that uses the speech unimodal feature and the expression unimodal feature as two inputs of a Siamese network, calculates a similarity loss function of the two outputs, and extracts shared information between the two modalities; and

[0034] Extracting specific information of the speech unimodal feature and the expression unimodal feature through orthogonal constraints to obtain an orthogonal loss function;

[0035] The sentiment classification module performs sentiment classification based on the shared information and the unique information to obtain a total loss function, and obtains a final sentiment classification by minimizing the total loss function.

[0036] Preferably, the similarity loss function is:

[0037]

[0038] in, is the similarity loss function, represents a set of emotion categories, and denote the shared features of speech and expression modalities, (·)i represents the features of the i-th sentiment category, and |·| and ||·|| represent the L1 and L2 norms, respectively.

[0039] Preferably, extracting specific information of the speech unimodal feature and the expression unimodal feature through orthogonal constraints to obtain an orthogonal loss function includes:

[0040] Extracting corresponding unique information according to the voice unimodal feature and the expression unimodal feature respectively;

[0041] The orthogonal loss function is calculated based on the shared information and the unique information of the two modalities.

[0042] Preferably, the orthogonal loss function is:

[0043]

[0044] in, is the orthogonal loss function, is a unique feature of speech, It is a characteristic of facial expression. Represents the dot product operation.

[0045] Preferably, obtaining the total loss function comprises:

[0046] Calculate the cross entropy loss function;

[0047] The total loss function is obtained according to the cross entropy loss function, the similarity loss function, and the orthogonal loss function.

[0048] Preferably, the cross entropy loss function is:

[0049]

[0050] in, is the cross entropy loss function, y i represents the sentiment label of the i-th sample, Represents the sentiment classification module C M (·) The predicted label for the i-th sample.

[0051] Preferably, the total loss function is:

[0052]

[0053] in, is the total loss function, and α represents a hyperparameter.

[0054] In a third aspect, an embodiment of the present disclosure further provides an electronic device, the electronic device comprising:

[0055] a memory storing executable instructions;

[0056] A processor runs the executable instructions in the memory to implement the power personnel emotion recognition method.

[0057] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the method for identifying the emotions of power personnel is implemented.

[0058] Its beneficial effects are:

[0059] (1) A Siamese network structure is constructed using two three-dimensional convolutional neural networks (3D-CNNs). The similarity loss function is used to map the speech and expression modal features to the same subspace to obtain the speech-expression shared features. This can effectively reduce the distribution gap between the two modalities caused by the complex and high-noise working environment of power workers.

[0060] (2) 3D-CNN is used to obtain the unique features of speech and expression respectively, and the orthogonal constraint loss function is used to reduce the information redundancy between the two modalities caused by the complex and high-noise working environment of power workers;

[0061] (3) Use the classification module to integrate the shared features and unique features of speech expressions and realize the classification of emotions.

[0062] The methods and apparatus of the present invention have other features and advantages that will be apparent from or will be described in detail in the accompanying drawings and subsequent detailed descriptions incorporated herein, which together serve to explain the specific principles of the invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] The above and other objects, features and advantages of the present invention will become more apparent through a more detailed description of exemplary embodiments of the present invention with reference to the accompanying drawings, wherein like reference numerals generally represent like components throughout the exemplary embodiments of the present invention.

[0064] Figure 1 A flowchart showing the steps of a method for identifying emotions of power personnel according to an embodiment of the present invention is shown.

[0065] Figure 2 A schematic diagram of a method for identifying emotions of power personnel according to an embodiment of the present invention is shown.

[0066] Figure 3A schematic diagram showing the results of each emotion category of the method according to one embodiment of the present invention on the BAUM-1s dataset.

[0067] Figure 4 A schematic diagram showing the results of each emotion category of the method on the RML dataset according to an embodiment of the present invention is shown.

[0068] Figure 5 A schematic diagram showing the results of each emotion category of the method according to one embodiment of the present invention on the eNTERFACE05 dataset.

[0069] Figure 6 A schematic diagram of a power personnel emotion recognition device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0070] The preferred embodiments of the present invention will be described in more detail below. Although the preferred embodiments of the present invention are described below, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein.

[0071] To facilitate understanding of the solutions and effects of the embodiments of the present invention, four specific application examples are given below. Those skilled in the art should understand that these examples are only for facilitating understanding of the present invention, and any specific details thereof are not intended to limit the present invention in any way.

[0072] Example 1

[0073] Figure 1 A flowchart showing the steps of a method for identifying emotions of power personnel according to an embodiment of the present invention is shown.

[0074] like Figure 1 As shown, the method for identifying the emotion of electric power personnel includes: step 101, extracting the voice unimodal features and the expression unimodal features respectively; step 102, taking the voice unimodal features and the expression unimodal features as the two inputs of the Siamese network respectively, calculating the similarity loss function of the two outputs, and extracting the shared information of the two modalities; step 103, extracting the specific information of the voice unimodal features and the expression unimodal features through orthogonal constraints, and obtaining the orthogonal loss function; step 104, performing emotion classification according to the shared information and the specific information, and obtaining the total loss function; step 105, obtaining the final emotion classification by minimizing the total loss function.

[0075] In one example, the similarity loss function is:

[0076]

[0077] in, is the similarity loss function, represents a set of emotion categories, and denote the shared features of speech and expression modalities, (·) i represents the features of the i-th sentiment category, and |·| and ||·|| represent the L1 and L2 norms, respectively.

[0078] In one example, specific information of speech unimodal features and expression unimodal features is extracted through orthogonal constraints, and the orthogonal loss function is obtained including:

[0079] Extracting corresponding unique information based on speech unimodal features and expression unimodal features respectively;

[0080] An orthogonal loss function is calculated based on the shared information and the unique information of the two modalities.

[0081] In one example, the orthogonal loss function is:

[0082]

[0083] in, is the orthogonal loss function, is a unique feature of speech, It is a characteristic of facial expression. Represents the dot product operation.

[0084] In one example, obtaining the total loss function includes:

[0085] Calculate the cross entropy loss function;

[0086] According to the cross entropy loss function, similarity loss function and orthogonal loss function, the total loss function is obtained.

[0087] In one example, the cross entropy loss function is:

[0088]

[0089] in, is the cross entropy loss function, y i represents the sentiment label of the i-th sample, Represents the sentiment classification module C M (·) The predicted label for the i-th sample.

[0090] In one example, the total loss function is:

[0091]

[0092] in, is the total loss function, and α represents a hyperparameter.

[0093] Figure 2 A schematic diagram of a method for identifying emotions of power personnel according to an embodiment of the present invention is shown.

[0094] Specifically, if Figure 2 As shown in the figure, this method mainly includes speech Mel-frequency cepstral coefficients (MFCC), speech expression shared feature extraction module Speech-specific feature extraction module F S Sp (·), expression-specific feature extraction module Sentiment Classification Module C M (·), similarity loss function L Sim , orthogonal loss function L Ort And the classification loss function L Cla .

[0095] Utilize voice expression sharing feature extraction module And the similarity loss function L Sim The speech and expression modal features are mapped to the same subspace to obtain the speech expression shared features, which effectively reduces the distribution gap between the two modalities caused by the complex and high noise working environment of power workers; the speech-specific feature extraction module F is used to extract the speech expression shared features. S Sp (·) and expression-specific feature extraction module Acquire the unique features of speech and expression respectively, and use the orthogonal loss function L Ort To reduce the information redundancy between the two modes caused by the complex and high noise working environment of power workers; sentiment classification module C M (·) and the classification loss function L Cla Realize the integration and classification of emotions.

[0096] Speech MFCCs based on cepstrum are consistent with human auditory habits and are widely used in speech emotion recognition. Therefore, this method also uses speech MFCCs to process speech modalities. Specifically, a 64-filter bank is used, one frame is selected every 16 frames, and a 64×64 speech MFCC is extracted. In addition, to simulate the RGB channels of the image, the first-order and second-order differences of the speech MFCCs are usually solved, where the difference formula is defined as:

[0097]

[0098] Among them, d t represents the first-order difference of the MFCC of the t-th frame speech, c t+n represents the MFCC of the speech of frame t+n, N represents the regression window, which is usually 2. t+n and c t-nReplace with d t+n and d t-n , the second-order difference of speech MFCC can be calculated using this formula. Finally, the RGB channels of speech MFCC, first-order speech MFCC, and second-order speech MFCC imitation image are input into the speech expression shared feature extraction module. and speech-specific feature extraction module F S Sp (·).

[0099] Voice expression shared feature extraction module It contains two 3D-CNNs with the same structure, each of which contains eight convolutional layers (Conv1a, Conv2a, Conv3a, Conv3b, Conv4a, Conv4b, Conv5a, Conv5b), 5 maximum pooling layers (Pool1, Pool2, Pool3, Pool4, Pool5) and two fully connected layers (FC6, FC7). These two 3D-CNNs construct a Siamese network structure through weight sharing. Unlike traditional neural networks that only contain one input and one network, Siamese networks usually contain two inputs and apply two neural networks to map the two inputs into the feature space. Then, a similarity loss function is used to explore the similarity between the two inputs. The similarity loss function is:

[0100]

[0101] Obviously, by minimizing the above formula, we can optimize the two 3D-CNNs in the Siamese structure and obtain the shared features of the speech modality. Shared features with emoji modalities In addition, voice expression shares feature extraction module And the similarity loss function L Sim Mapping speech and facial expression modal features to the same subspace can effectively reduce the distribution gap between the two modalities caused by the complex and noisy working environment of power workers.

[0102] In addition to designing a voice expression shared feature extraction module And the similarity loss function L Sim Obtaining shared features of speech modalities Shared features with emoji modalities Unique features of speech and facial expressions can complement shared features of speech modality Shared features with emoji modalities Thus, the performance of emotion recognition is improved. Therefore, the speech-specific feature extraction module F S Sp(·) Take speech MFCC as input and apply 3D-CNN to obtain speech-specific features Similarly, the expression-specific feature extraction module Using facial expressions as input, 3D-CNN is also applied to obtain facial expression-specific features.

[0103] To reduce the shared features of speech modes Shared features of expression modalities Unique features of speech and facial features The information redundancy between them can be forced to constrain these features through the orthogonal constraint loss function. Among them, the orthogonal constraint loss function is:

[0104]

[0105] By minimizing the above formula, the shared features of speech modalities can be effectively reduced. and speech-specific features Shared features of expression modalities and facial features There is information redundancy between them.

[0106] Utilize voice expression sharing feature extraction module Similarity loss function L Sim , Speech-specific feature extraction module F S Sp (·), Expression-specific feature extraction module And the orthogonality constraint loss function L Ort , not only can we obtain the shared features of speech modalities Shared features of expression modalities Unique features of speech and facial features It also reduces the distribution gap and information redundancy between the two modalities caused by the complex and high noise working environment of power workers. Then, these features are fused and input into the sentiment classification module C M (·) to realize emotion recognition. Among them, the emotion classification module C M (·) A three-layer multi-layer perceptron (MLP) is used, and the last layer uses the softmax function to output the classification result. In addition, the classification uses the classic cross entropy loss function:

[0107]

[0108] Finally, the total loss function is:

[0109]

[0110] Obviously, by minimizing the total loss function, not only can sentiment classification be achieved, but the distribution gap between the two modalities caused by the complex and high noise working environment of power personnel can also be effectively reduced, which can effectively improve the performance of emotion recognition of power personnel.

[0111] Figure 3 A schematic diagram showing the results of each emotion category of the method according to one embodiment of the present invention on the BAUM-1s dataset.

[0112] Figure 4 A schematic diagram showing the results of each emotion category of the method on the RML dataset according to an embodiment of the present invention is shown.

[0113] Figure 5 A schematic diagram showing the results of each emotion category of the method according to one embodiment of the present invention on the eNTERFACE05 dataset.

[0114] The BAUM-1s, RML, and eNTERFACE05 datasets were selected to evaluate the performance of the proposed power personnel emotion recognition method that integrates shared and unique features. Six discrete emotions were classified, namely fear, disgust, happiness, anger, sadness, and surprise. Figure 3-Figure 5 The proposed method for power personnel emotion recognition, which integrates shared and unique features, is presented for each emotion category on three datasets. "Fe," "Di," "Ha," "An," "Sa," and "Su" represent fear, disgust, happiness, anger, sadness, and surprise, respectively. Furthermore, the proposed method for power personnel emotion recognition, which integrates shared and unique features, is compared with commonly used hybrid deep fusion models and correlated feature fusion models. The results are shown in Table 1.

[0115] Table 1

[0116]

[0117] The hybrid deep fusion model uses a deep learning algorithm to extract speech and facial expressions, then directly fuses the speech and facial expressions using a deep belief network, and finally performs classification using a support vector machine. The correlated feature fusion model also uses a deep learning algorithm to extract features, then analyzes speech and facial expressions using a canonical correlation analysis algorithm, and finally fuses the analyzed features for classification. Comparative experimental results show that this method achieves higher recognition rates than both the hybrid deep fusion model and the correlated feature fusion model. The reason for this is as follows: the hybrid deep fusion model directly fuses features without considering the correlation between speech and facial expressions. While the correlated feature fusion model considers the correlation between speech and facial expressions, it does not account for the distribution gap between the two modalities. In contrast, this method utilizes the Siamese structure and similarity loss function to effectively account for the distribution gap between the two modalities caused by the complex and noisy working environment of power workers, and employs orthogonal constraints to reduce the information redundancy between the two modalities caused by the complex and noisy working environment of power workers. As a result, this method achieves higher recognition rates than both the hybrid deep fusion model and the correlated feature fusion model.

[0118] Example 2

[0119] Figure 6 A block diagram of a device for identifying emotions of power personnel according to an embodiment of the present invention is shown.

[0120] like Figure 6 As shown, the power personnel emotion recognition device includes:

[0121] The unimodal feature extraction module extracts speech unimodal features and expression unimodal features respectively;

[0122] The feature extraction module uses the speech unimodal features and the expression unimodal features as the two inputs of the Siamese network, calculates the similarity loss function of the two outputs, and extracts the shared information of the two modalities; and

[0123] The specific information of speech unimodal features and expression unimodal features is extracted through orthogonal constraints to obtain the orthogonal loss function;

[0124] The sentiment classification module performs sentiment classification based on shared information and unique information, obtains the total loss function, and obtains the final sentiment classification by minimizing the total loss function.

[0125] In one example, the similarity loss function is:

[0126]

[0127] in, is the similarity loss function, represents a set of emotion categories, and denote the shared features of speech and expression modalities, (·) i represents the features of the i-th sentiment category, and |·| and ||·|| represent the L1 and L2 norms, respectively.

[0128] In one example, specific information of speech unimodal features and expression unimodal features is extracted through orthogonal constraints, and the orthogonal loss function is obtained including:

[0129] Extracting corresponding unique information based on speech unimodal features and expression unimodal features respectively;

[0130] An orthogonal loss function is calculated based on the shared information and the unique information of the two modalities.

[0131] In one example, the orthogonal loss function is:

[0132]

[0133] in, is the orthogonal loss function, is a unique feature of speech, It is a characteristic of facial expression. Represents the dot product operation.

[0134] In one example, obtaining the total loss function includes:

[0135] Calculate the cross entropy loss function;

[0136] According to the cross entropy loss function, similarity loss function and orthogonal loss function, the total loss function is obtained.

[0137] In one example, the cross entropy loss function is:

[0138]

[0139] in, is the cross entropy loss function, y i represents the sentiment label of the i-th sample, Represents the sentiment classification module C M (·) The predicted label for the i-th sample.

[0140] In one example, the total loss function is:

[0141]

[0142] in, is the total loss function, and α represents a hyperparameter.

[0143] Specifically, this system primarily consists of a field layer, a communication network layer, and a control layer. The field layer utilizes devices such as cameras and audio sensors to collect real-time facial expressions and voice information from power workers. This information is then transmitted to the fiber optic ring network via fiber optic switches at the communication network layer. The fiber optic ring network then inputs this information into the control layer's master control module. This master control module utilizes a proposed power worker emotion recognition method that integrates shared and unique features to identify the worker's emotional state online. This feedback is fed back to the emotion display module to help managers analyze the worker's emotional state, thereby reducing operational violations and errors and safeguarding national power safety. Furthermore, because the method employed by this system's core control module effectively reduces the distribution gap and information redundancy between voice and expression modalities, the system can effectively improve the performance of power worker emotion recognition.

[0144] Example 3

[0145] The present disclosure provides an electronic device comprising: a memory storing executable instructions; and a processor executing the executable instructions in the memory to implement the above-mentioned method for emotion recognition of electric power personnel.

[0146] An electronic device according to an embodiment of the present disclosure includes a memory and a processor.

[0147] The memory is used to store non-transitory computer-readable instructions. Specifically, the memory may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), a hard disk, flash memory, etc.

[0148] The processor may be a central processing unit (CPU) or other form of processing unit having data processing capability and / or instruction execution capability, and may control other components in the electronic device to perform desired functions. In one embodiment of the present disclosure, the processor is used to execute the computer-readable instructions stored in the memory.

[0149] Those skilled in the art should understand that in order to solve the technical problem of how to obtain a good user experience, this embodiment may also include well-known structures such as a communication bus and an interface, and these well-known structures should also be included in the scope of protection of this disclosure.

[0150] For detailed description of this embodiment, please refer to the corresponding description in the aforementioned embodiments, which will not be repeated here.

[0151] Example 4

[0152] An embodiment of the present disclosure provides a computer-readable storage medium storing a computer program, which implements the power personnel emotion recognition method when executed by a processor.

[0153] According to an embodiment of the present disclosure, a computer-readable storage medium stores non-transitory computer-readable instructions, which, when executed by a processor, execute all or part of the steps of the aforementioned methods of the embodiments of the present disclosure.

[0154] The above-mentioned computer-readable storage media include, but are not limited to, optical storage media (e.g., CD-ROMs and DVDs), magneto-optical storage media (e.g., MOs), magnetic storage media (e.g., magnetic tapes or mobile hard disks), media with built-in rewritable non-volatile memory (e.g., memory cards), and media with built-in ROM (e.g., ROM cartridges).

[0155] Those skilled in the art should understand that the above description of the embodiments of the present invention is only for the purpose of illustrative purposes only to illustrate the beneficial effects of the embodiments of the present invention, and is not intended to limit the embodiments of the present invention to any given examples.

[0156] While various embodiments of the present invention have been described above, the above description is intended to be illustrative, not exhaustive, and not limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments.

Claims

1. A method for identifying the emotions of power personnel, characterized in that: include: Extract speech unimodal features and expression unimodal features respectively; The speech unimodal feature and the expression unimodal feature are respectively used as two inputs of the Siamese network, and a similarity loss function of the two outputs is calculated to extract the shared information of the two modalities; Extracting unique information of the speech unimodal feature and the expression unimodal feature through orthogonal constraints to obtain an orthogonal loss function; Perform sentiment classification based on the shared information and the unique information to obtain a total loss function; Obtain the final sentiment classification by minimizing the total loss function; Wherein, the similarity loss function is: in, is the similarity loss function, represents a set of emotion categories, and denote the shared features of speech and expression modalities, (·) i represents the features of the i-th sentiment category, |·| and ||·|| represent the L1 and L2 norms respectively; The orthogonal loss function is obtained by extracting the unique information of the speech unimodal feature and the expression unimodal feature through orthogonal constraints, and comprising: Extracting corresponding unique information according to the voice unimodal feature and the expression unimodal feature respectively; Calculating the orthogonal loss function based on the shared information and the unique information of the two modalities; Among them, obtaining the total loss function includes: Calculate the cross entropy loss function; The total loss function is obtained according to the cross entropy loss function, the similarity loss function, and the orthogonal loss function.

2. The method for identifying the emotion of electric power personnel according to claim 1, wherein: The orthogonal loss function is: in, is the orthogonal loss function, is a unique feature of speech, It is a unique feature of facial expression. T Represents the dot product operation.

3. The method for identifying the emotion of electric power personnel according to claim 1, wherein: The cross entropy loss function is: in, is the cross entropy loss function, y i represents the sentiment label of the i-th sample, Represents the sentiment classification module The predicted label for the i-th sample.

4. The method for identifying the emotion of electric power personnel according to claim 1, wherein: The total loss function is: in, is the total loss function, and α represents a hyperparameter.

5. A power personnel emotion recognition device, characterized in that: The device is used to implement the power personnel emotion recognition method according to any one of claims 1 to 4, comprising: The unimodal feature extraction module extracts speech unimodal features and expression unimodal features respectively; a feature extraction module that uses the speech unimodal feature and the expression unimodal feature as two inputs of a Siamese network, calculates a similarity loss function of the two outputs, and extracts shared information between the two modalities; and Extracting unique information of the speech unimodal feature and the expression unimodal feature through orthogonal constraints to obtain an orthogonal loss function; The sentiment classification module performs sentiment classification based on the shared information and the unique information to obtain a total loss function, and obtains a final sentiment classification by minimizing the total loss function.

6. An electronic device, characterized in that: The electronic device comprises: a memory storing executable instructions; A processor, wherein the processor runs the executable instructions in the memory to implement the power personnel emotion recognition method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the power personnel emotion recognition method according to any one of claims 1 to 4.

Images