Ship driver emotion recognition method, system and device and medium

Abstract

The invention discloses an emotion recognition method, system and device for ship drivers and a medium, and an emotion recognition model in the method is obtained through the following steps: preprocessing face original data and voice original data to obtain face training data and voice training data; inputting the stage emotion training data into a stage emotion network framework for feature extraction to obtain stage emotion features; inputting the face training data into a face network framework for feature extraction to obtain face emotion features, and inputting the voice training data into a voice network framework for feature extraction to obtain voice emotion features; and according to the stage emotional features, the face emotional features and the voice emotional features, jointly training an initialized emotional recognition model. According to the method, the robustness in the emotion recognition process can be effectively improved, and the recognition accuracy of ship driver emotion recognition is improved. The invention relates to the technical field of artificial intelligence.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for identifying emotions of ship drivers. Background Art

[0002] At present, traditional driver identification methods usually realize driver emotion recognition based on unimodal data. This method is mostly used for emotion recognition of vehicle drivers. However, due to complex environmental factors such as complex lighting conditions in the ship's cab, mutual occlusion between ship personnel, and shaking during ship driving, the robustness of emotion recognition based on unimodal data is poor, and the recognition accuracy of ship driver emotion recognition is not satisfactory.

[0003] Therefore, the problems existing in the prior art still need to be solved and optimized. Summary of the invention

[0004] The purpose of the present invention is to solve one of the technical problems existing in the related art to at least a certain extent.

[0005] To this end, an object of an embodiment of the present invention is to provide a method for identifying emotions of ship drivers, which can effectively improve the robustness of the emotion recognition process and improve the recognition accuracy of the emotion recognition of ship drivers.

[0006] Another object of an embodiment of the present application is to provide an emotion recognition system for ship drivers.

[0007] In order to achieve the above technical objectives, the technical solutions adopted by the embodiments of the present application include:

[0008] In a first aspect, an embodiment of the present application provides a method for identifying emotions of a ship driver, comprising:

[0009] Obtain the facial data, voice data and staged emotional state of the ship driver to be emotion recognized;

[0010] Inputting the face data, the voice data and the staged emotional state into a trained emotion recognition model to obtain an emotion recognition result of the ship driver;

[0011] The emotion recognition model includes a parallel voice network framework, a face network framework and a stage emotion network framework, and the trained emotion recognition model is obtained by the following steps:

[0012] Obtain the target ship driver's stage emotion training data, face raw data and voice raw data;

[0013] Performing a first preprocessing on the original face data to obtain face training data, and performing a second preprocessing on the original voice data to obtain voice training data;

[0014] Inputting the stage emotion training data into the stage emotion network framework for feature extraction to obtain stage emotion features, wherein the stage emotion features are used to characterize the weights of multiple emotion types corresponding to the target ship driver;

[0015] Inputting the face training data into the face network framework to extract features and obtain face emotion features, and inputting the voice training data into the voice network framework to extract features and obtain voice emotion features;

[0016] The initialized emotion recognition model is jointly trained according to the stage emotion features, the facial emotion features and the voice emotion features to obtain the trained emotion recognition model.

[0017] In addition, the method according to the above embodiment of the present application may also have the following additional technical features:

[0018] Furthermore, in one embodiment of the present application, the first preprocessing of the original face data to obtain face training data includes:

[0019] Get a random occlusion rectangle and brightness level;

[0020] Cleaning and editing the original face data to obtain intermediate face data;

[0021] Based on the occlusion rectangle and the brightness level, data enhancement is performed on the intermediate face data to obtain the face training data;

[0022] The second preprocessing of the original speech data to obtain speech training data includes:

[0023] Performing voice data cleaning on the original voice data to obtain voice intermediate data;

[0024] The speech intermediate data is clipped to obtain the speech training data.

[0025] Further, in one embodiment of the present application, the stage emotion network framework includes a weighted network module and a transformation network module, and the stage emotion training data is input into the stage emotion network framework for feature extraction to obtain the stage emotion feature, including:

[0026] Inputting the stage emotion training data into the weighted network module for weighted summation to obtain intermediate emotion features;

[0027] The intermediate emotion features are input into the transformation network module for nonlinear transformation to obtain the stage emotion features.

[0028] Further, in one embodiment of the present application, the step of inputting the face training data into the face network framework for feature extraction to obtain face emotion features includes:

[0029] Extracting temporal attention features from the face training data to obtain temporal features of the face;

[0030] Performing spatial attention feature extraction on the temporal features of the human face to obtain temporal and spatial features of the human face;

[0031] The face spatiotemporal features are sampled at multiple time scales to obtain the face emotion features.

[0032] Further, in one embodiment of the present application, the multi-time scale upsampling of the face spatiotemporal features to obtain the face emotion features includes:

[0033] Performing short-time-scale sampling on the face spatiotemporal features to obtain short-time-scale face features;

[0034] Performing mesoscale sampling on the face spatiotemporal features to obtain mesoscale face features;

[0035] Performing long-time-scale sampling on the face spatiotemporal features to obtain long-time-scale face features;

[0036] According to the short-time scale facial features and the medium-time scale facial features, channel splicing is performed on the long-time scale facial features to obtain the facial emotion features;

[0037] Among them, the short-time scale facial features, the medium-time scale facial features and the long-time scale facial features are aligned with each other in the time dimension.

[0038] Further, in one embodiment of the present application, the speech network framework includes a convolutional network module and a time recurrent network module, and the speech training data is input into the speech network framework for feature extraction to obtain speech emotion features, including:

[0039] Determining target speech features based on the speech training data;

[0040] Inputting the target speech feature into the convolutional network module to extract local features and obtain intermediate speech features;

[0041] The intermediate speech features are input into the time recurrent network module to extract time-dependent features to obtain the speech emotion features.

[0042] Further, in one embodiment of the present application, the step of jointly training the initialized emotion recognition model according to the stage emotion feature, the facial emotion feature and the voice emotion feature to obtain the trained emotion recognition model includes:

[0043] According to the stage emotion features, feature fusion and classification are performed on the facial emotion features and the voice emotion features to obtain a target fusion classification vector;

[0044] According to the target fusion classification vector, the parameters of the initialized emotion recognition model are updated to obtain the trained emotion recognition model.

[0045] In a second aspect, an embodiment of the present application provides an emotion recognition system for ship drivers, comprising:

[0046] The first processing unit is used to obtain the face data, voice data and staged emotional state of the ship driver to be emotion recognized;

[0047] A second processing unit is used to input the face data, the voice data and the staged emotional state into the trained emotion recognition model to obtain an emotion recognition result of the ship driver;

[0048] The emotion recognition model includes a parallel voice network framework, a face network framework and a stage emotion network framework, and the trained emotion recognition model is obtained by the following steps:

[0049] Obtain the target ship driver's stage emotion training data, face raw data and voice raw data;

[0050] Performing a first preprocessing on the original face data to obtain face training data, and performing a second preprocessing on the original voice data to obtain voice training data;

[0051] Inputting the stage emotion training data into the stage emotion network framework for feature extraction to obtain stage emotion features, wherein the stage emotion features are used to characterize the weights of multiple emotion types corresponding to the target ship driver;

[0052] Inputting the face training data into the face network framework to extract features and obtain face emotion features, and inputting the voice training data into the voice network framework to extract features and obtain voice emotion features;

[0053] The initialized emotion recognition model is jointly trained according to the stage emotion features, the facial emotion features and the voice emotion features to obtain the trained emotion recognition model.

[0054] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0055] at least one processor;

[0056] at least one memory for storing at least one program;

[0057] When the at least one program is executed by the at least one processor, the at least one processor implements the method of the first aspect described above.

[0058] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores a program executable by a processor, and the program executable by the processor is used to implement the method of the first aspect when executed by the processor.

[0059] The advantages and benefits of the present application will be partially given in the following description, and partially become apparent from the following description, or be understood through the practice of the present application:

[0060] The embodiment of the present application discloses an emotion recognition method, system, device and medium for ship driver, wherein the emotion recognition method obtains face data, voice data and stage emotion state of the ship driver to be emotion recognized; inputs the face data, voice data and stage emotion state into a trained emotion recognition model to obtain the emotion recognition result of the ship driver; wherein the emotion recognition model includes a parallel voice network framework, a face network framework and a stage emotion network framework, and the trained emotion recognition model is obtained by the following steps: obtaining the stage emotion training data, face raw data and voice raw data of the target ship driver; performing a first preprocessing on the face raw data to obtain the face raw data; The method comprises the following steps: inputting the stage emotion training data into the face network framework for feature extraction to obtain the face emotion features, and performing a second preprocessing on the original speech data to obtain the speech training data; inputting the stage emotion training data into the stage emotion network framework for feature extraction to obtain the stage emotion features, and the stage emotion features are used to characterize the weights of multiple emotion types corresponding to the target ship driver; inputting the face training data into the face network framework for feature extraction to obtain the face emotion features, and inputting the speech training data into the speech network framework for feature extraction to obtain the speech emotion features; and jointly training the initialized emotion recognition model according to the stage emotion features, the face emotion features and the speech emotion features to obtain the trained emotion recognition model. This emotion recognition method can effectively improve the robustness of the emotion recognition model in the process of emotion recognition of ship drivers by performing emotion recognition on multimodal data such as facial data, voice data and staged emotional states; in addition, this emotion method is based on the stage emotion features, facial emotion features and voice emotion features that characterize the weights of multiple emotion types of ship drivers, and jointly trains the emotion recognition model, which can enable the trained emotion recognition model to pay targeted attention to ship drivers in poor emotional states, thereby improving the recognition accuracy of the emotion recognition model in identifying the emotions of ship drivers. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the embodiments of the present application or the drawings of the related technical solutions in the prior art are introduced below. It should be understood that the drawings introduced below are only for the convenience of clearly expressing some embodiments of the technical solutions of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0062] Figure 1 A schematic diagram of a flow chart of a method for identifying emotions of ship drivers provided in an embodiment of the present application;

[0063] Figure 2A schematic diagram of a training process of an emotion recognition model provided in an embodiment of the present application;

[0064] Figure 3 A schematic diagram of the structure of an emotion recognition system for ship drivers provided in an embodiment of the present application;

[0065] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] Embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present application, and are not to be construed as limitations on the present application. For the step numbers in the following embodiments, they are only provided for the convenience of explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiment can be adaptively adjusted according to the understanding of those skilled in the art.

[0067] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0068] At present, traditional driver identification methods usually realize driver emotion recognition based on unimodal data. This method is mostly used for emotion recognition of vehicle drivers. However, due to complex environmental factors such as complex lighting conditions in the ship's cab, mutual occlusion between ship personnel, and shaking during ship driving, the robustness of emotion recognition based on unimodal data is poor, and the recognition accuracy of ship driver emotion recognition is not satisfactory.

[0069] Specifically, the driver emotion recognition method based on unimodal data can be roughly divided into driver emotion recognition based on facial expressions, driver emotion recognition based on voice signals, and driver emotion recognition based on physiological signals. Among them, for driver emotions based on physiological signals, the driver's physiological information is collected through equipment, and then machine learning or deep learning methods are used to process features and predict driver emotions. However, such methods usually require the driver to wear corresponding detection devices, which will affect the driver's behavior and mental state. Driver emotion recognition based on facial expressions adopts a non-contact detection approach. It only needs to use a video camera installed inside the cab to collect images, and use computer vision technology to extract the driver's facial emotional features to determine whether he is in a good emotional state. In addition, driver emotion recognition based on voice signals also has the advantage of being non-invasive. It only needs to collect voice data in the cockpit and distinguish emotions based on the emotional features in the voice.

[0070] These driver emotion recognition algorithms based on single-modal data are easily affected by interference and have poor robustness. They often cannot capture the contextual information in the ship's cab well in the complex driving environment of the ship (such as illumination changes, shaking blur, personnel occlusion, etc.), and the recognition accuracy of ship driver emotion recognition is not satisfactory.

[0071] In addition, existing emotion recognition methods mainly focus on detecting and identifying the driver's current emotions, and do not consider that the stage-by-stage emotional state is an important factor in the driver's emotion management and control (such as a bad emotional state may cause an individual to exhibit potentially dangerous behaviors). The recognition accuracy and reliability of driver emotion recognition are not high.

[0072] In view of this, an embodiment of the present invention provides an emotion recognition method for ship drivers, wherein the method can effectively improve the robustness of the emotion recognition model in the process of emotion recognition for ship drivers by performing emotion recognition on multimodal data such as facial data, voice data and stage-by-stage emotional state; in addition, the emotion method is based on stage emotion features, facial emotion features and voice emotion features that characterize the weights of multiple emotion types of ship drivers, and jointly trains the emotion recognition model, so that the trained emotion recognition model can achieve targeted attention to ship drivers in poor emotional states, thereby improving the recognition accuracy of the emotion recognition model in recognizing the emotions of ship drivers.

[0073] In addition, this method also performs data enhancement on facial data based on occlusion rectangles and brightness levels, which can enable the obtained facial training data to correspond to complex ship driving environments such as lighting changes and occlusions, and effectively increase the robustness of the emotion recognition model in complex ship driving.

[0074] Reference Figure 1 and Figure 2 In an embodiment of the present application, a method for identifying emotions of a ship driver includes:

[0075] Step 110, obtaining face data, voice data and staged emotional state of the ship driver to be emotion recognized;

[0076] Step 120: input the face data, the voice data and the staged emotional state into the trained emotion recognition model to obtain the emotion recognition result of the ship driver;

[0077] In an embodiment of the present application, facial data can be obtained by photographing the facial image of the ship driver in the ship's cab with a camera device, and voice data can be obtained by collecting the sound signal of the ship driver in the ship's cab with a sound collection device; the stage emotional state is used to characterize the emotional state of the ship driver at the current stage, which can be specifically collected through the Implicit Emotion Scale (IPANAS) and other methods, which will not be elaborated in this application.

[0078] It can be understood that after obtaining facial data, voice data and stage-by-stage emotional state, the facial data, voice data and stage-by-stage emotional state can be input together into a trained emotion recognition model for emotion recognition, and the ship driver emotion recognition result output by the trained emotion recognition model can be obtained. The ship driver emotion recognition result is used to characterize the emotional state of the ship driver at the current stage.

[0079] The emotion recognition model includes a parallel voice network framework, a face network framework and a stage emotion network framework, and the trained emotion recognition model is obtained by the following steps:

[0080] Step 130: Acquire the stage emotion training data, face original data and voice original data of the target ship driver;

[0081] In an embodiment of the present application, stage emotion training data, original face data and original voice data can be multimodal data for emotion recognition model training, which can be a data set that has been publicly available on the Internet, or historical multimodal data collected by corresponding equipment during ship driving. The examples in this application are for illustration only and do not limit the data sources of stage emotion training data, original face data and original voice data.

[0082] Step 140: performing a first preprocessing on the original face data to obtain face training data, and performing a second preprocessing on the original voice data to obtain voice training data;

[0083] In an embodiment of the present application, after obtaining the original face data and the original voice data, the original face data and the original voice data can be preprocessed respectively to convert the original face data and the original voice data into face training data and voice training data that can be understood and processed by the neural network. The specific preprocessing method may include data cleaning, data normalization, data enhancement and the like.

[0084] In some embodiments, the step 140 of performing a first preprocessing on the original face data to obtain face training data includes:

[0085] A1. Get a random occlusion rectangle and brightness level;

[0086] A2. Cleaning and editing the original face data to obtain intermediate face data;

[0087] A3. Based on the occlusion rectangle and the brightness level, perform data enhancement on the intermediate face data to obtain the face training data;

[0088] In an embodiment of the present application, the occlusion rectangle obtained in step A1 can be a rectangle of random size, which is used to randomly occlude each facial image in the intermediate face data, and the random brightness level is used to randomly adjust the lighting changes of each facial image in the intermediate face data.

[0089] It can be understood that step A2 can first remove the facial images with blurred images or incorrect expressions in the original facial data, and then edit each facial image in the removed original facial data to obtain intermediate facial data, which can generally include several continuous expression frames of the driver with different facial emotions.

[0090] It should be noted that, for a certain face image in the intermediate face data, step A3 can be any data enhancement method, such as using a randomly sized occluding rectangle to occlude the random position of the driver's face in the face image, using a random brightness level to process the lighting changes in the face image, using rotation, scaling, etc. to randomly distort the face image, and the same is true for the remaining face images, so as to obtain face training data.

[0091] The step 140 of performing a second preprocessing on the original speech data to obtain speech training data includes:

[0092] B1. Cleaning the original voice data to obtain intermediate voice data;

[0093] B2. Edit the speech intermediate data to obtain the speech training data.

[0094] In an embodiment of the present application, step B1 can be to remove the speech signals with smaller sound and fuzzy sound in the original speech data. Specifically, a loudness threshold and a noise threshold can be preset. The size of the loudness threshold and the noise threshold can be flexibly set according to the actual situation, and then the size of each speech signal in the original speech data is compared based on the loudness threshold and the noise threshold, and the speech signals with smaller sound and fuzzy sound in the original speech data are removed based on the size comparison result, so as to obtain the speech intermediate data. There are many specific implementation methods, which will not be repeated in this application. It can be understood that step B2 can be to clip each speech signal in the intermediate speech data based on the preset speech length to obtain a number of speech clipping signals that meet the speech length, and then obtain the speech training data by integrating all the speech clipping signals.

[0095] Step 150: input the stage emotion training data into the stage emotion network framework for feature extraction to obtain stage emotion features, wherein the stage emotion features are used to characterize the weights of multiple emotion types corresponding to the target ship driver;

[0096] In an embodiment of the present application, a stage emotion network framework can be constructed based on a deep neural network (DNN). The stage emotion training data specifically input into the stage emotion network framework can be the stage emotion state score of the ship driver. The stage emotion feature output by the stage emotion network framework is the personalized emotion feature of the ship driver. The stage emotion feature can be represented by a feature vector, and each vector dimension in the feature vector is used to characterize the weight corresponding to one of the emotion types of the target ship driver.

[0097] In some embodiments, the stage emotion network framework includes a weighted network module and a transformation network module, and the step 150 of inputting the stage emotion training data into the stage emotion network framework for feature extraction to obtain the stage emotion feature includes:

[0098] C1. Inputting the stage emotion training data into the weighted network module for weighted summation to obtain intermediate emotion features;

[0099] C2. Inputting the intermediate emotion feature into the transformation network module for nonlinear transformation to obtain the stage emotion feature.

[0100] In an embodiment of the present application, the weighted network module may be a fully connected layer in a DNN network, and step C1 may be inputting the stage emotion training data into the stage emotion network framework, and performing weighted summation on all neuron outputs of the previous layer through the weight coefficient of the fully connected layer to obtain the intermediate emotion features; step C2 may be performing a nonlinear transformation on the intermediate emotion features based on an activation function layer (such as a ReLu activation function layer) in the DNN network to obtain a feature vector containing information in multiple dimensions.

[0101] It can be understood that in practical applications, step C1 and step C2 can be executed multiple times in a loop, and the specific number of loop executions can be determined according to the actual situation, and the feature vector obtained by the last loop execution is determined as the final stage emotion feature. Exemplarily, if the stage emotion feature is a feature vector of size (1, 7), the feature vector is a personalized emotion feature corresponding to the ship driver, in which each vector dimension can represent the weight of one of the emotion types currently corresponding to the ship driver, and the specific emotion types can include anger, pain, panic, sadness, calmness, happiness, surprise, etc. The examples in this application are for illustration only and are not intended to limit specific emotion types.

[0102] Step 160: input the face training data into the face network framework to extract features to obtain face emotion features, and input the voice training data into the voice network framework to extract features to obtain voice emotion features;

[0103] In an embodiment of the present application, a face network framework can be constructed by a three-dimensional convolutional neural network (3DCNN), and specifically can be constructed by adding a 3D MobileNetV2 convolutional neural network structure and a temporal pyramid (TPN) network structure with a spatiotemporal attention mechanism. The features in the face training data are extracted by adding the MobileNetV2 convolutional neural network structure and the temporal pyramid (TPN) network structure with a spatiotemporal attention mechanism to obtain facial emotion features. In addition, the voice network framework can be a CNN-LSTM network structure. The features in the voice training data are extracted by the CNN-LSTM network structure to obtain voice emotion features.

[0104] In some embodiments, the step 160 of inputting the face training data into the face network framework for feature extraction to obtain facial emotion features includes:

[0105] D1. Extracting temporal attention features from the face training data to obtain temporal features of the face;

[0106] D2, extracting spatial attention features from the temporal features of the face to obtain spatiotemporal features of the face;

[0107] In an embodiment of the present application, for a certain facial image continuous frame in the face training data, the temporal attention feature extraction in step D1 can be to extract the time step feature in each frame image from the facial image continuous frame, and calculate the temporal attention weight for each time step of the facial image continuous frame, and then apply the temporal attention weight to the time step feature, so as to obtain the facial temporal feature, which can highlight the facial emotion feature at the important time point (i.e., the emotion peak) in the facial image continuous frame. Specifically, for the ship driving environment, the lighting conditions in the ship's cab are relatively complex, and personnel often sway and occlude each other during the process of driving the ship. The emotion peak frame image in the facial image continuous frame has a short existence time. The embodiment of the present application can extract the temporal attention feature to enable the emotion recognition model to pay more attention to the emotion peak frame image in the facial image continuous frame, which helps to improve the recognition accuracy of emotion recognition.

[0108] It can be understood that the spatial attention feature extraction in step D2 is used to further extract the facial detail features of the driver in the facial image. The content of the spatial attention feature extraction is similar to the content of the aforementioned temporal attention feature extraction and can be simply deduced by analogy.

[0109] Exemplarily, for a 3D MobileNetV2 convolutional neural network structure with a spatiotemporal attention mechanism, step D1 can first extract the time step features of the continuous frames of the input facial image through a 3×3×3 sized 3D convolution layer, and then project the time step features to a lower dimensional time step feature representation through a 1×1 convolution layer; then, the time attention weight is calculated based on the lower dimensional time step feature representation, and the time attention weight is applied to the time step feature, so that the emotion recognition model pays more attention to the emotion peak frame image in the time series of the continuous frames of the facial image, thereby obtaining a time feature map (i.e., face time feature). After obtaining the time feature map, step D2 can first obtain the spatiotemporal features by performing a number of 3×3×3 sized 3D convolution layers to obtain the important information of the time feature map in the spatial dimension; then, the spatiotemporal features are globally averaged pooled in the time dimension, and the pooled features are fused to obtain the spatial attention map; then, the spatial attention weight corresponding to the spatial attention map is calculated through a 3D convolution layer, and the spatial attention weight is applied to the spatiotemporal features, thereby obtaining the spatiotemporal feature map (i.e., face spatiotemporal features).

[0110] D3. Perform multi-time scale sampling on the face spatiotemporal features to obtain the face emotion features.

[0111] Furthermore, the step D3 of performing multi-time scale upsampling on the face spatiotemporal features to obtain the face emotion features includes:

[0112] D31, performing short-time scale sampling on the face spatiotemporal features to obtain short-time scale face features;

[0113] D32, performing mesoscale sampling on the face spatiotemporal features to obtain mesoscale face features;

[0114] D33, performing long-time-scale sampling on the face spatiotemporal features to obtain long-time-scale face features;

[0115] D34, performing channel splicing on the long time scale facial features according to the short time scale facial features and the medium time scale facial features to obtain the facial emotion features;

[0116] In some embodiments, the multi-time scale upsampling in step D3 is used to extract time features of different time scales in the face spatiotemporal features. Specifically, step D31 may be to use a 1×1×1 sized 3D convolution kernel to perform a convolution operation on the face spatiotemporal features to obtain a short-time scale time feature, and then perform an upsampling operation on the short-time scale time feature so that the time dimension of the short-time scale face feature is consistent with the face spatiotemporal features of the original input; step D32 may be to use a 2×1×1 sized 3D convolution kernel to perform a convolution operation on the face spatiotemporal features to obtain a medium-time scale time feature, and then perform an upsampling operation on the medium-time scale time feature so that the time dimension of the medium-time scale face feature is consistent with the face spatiotemporal features of the original input; step D34 may be to use a 4×1×14 sized 3D convolution kernel to perform a convolution operation on the face spatiotemporal features to obtain a long-time scale time feature, and then perform an upsampling operation on the long-time scale time feature so that the time dimension of the long-time scale face feature is consistent with the face spatiotemporal features of the original input.

[0117] It can be understood that after making the facial features of different time scales consistent with the original input facial spatiotemporal features, the facial features of different time scales can be aligned with each other in the time dimension. At this time, the short-time scale facial features, medium-time scale facial features and long-time scale facial features can be spliced ​​in the channel dimension to obtain facial emotion features.

[0118] In some embodiments, the speech network framework includes a convolutional network module and a time recurrent network module, and the step 160 of inputting the speech training data into the speech network framework for feature extraction to obtain speech emotion features includes:

[0119] E1. Determine target speech features based on the speech training data;

[0120] E2. Inputting the target speech feature into the convolutional network module to extract local features and obtain intermediate speech features;

[0121] E3. Input the intermediate speech features into the time recurrent network module to extract time-dependent features to obtain the speech emotion features.

[0122] In an embodiment of the present application, step E1 may be to determine the prosodic features and spectral features of each speech signal in the speech training data as the required target speech features. Exemplarily, for a certain speech signal, its prosodic features may be obtained by constructing an autocorrelation function of the speech signal, and specifically, the fundamental frequency corresponding to the first local maximum value between a certain lowest frequency corresponding delay and the highest frequency in the autocorrelation function may be determined as the prosodic feature; and the spectral features may be obtained using Mel-frequency cepstral coefficients (MFCC), and there are many specific implementation methods, which will not be described in detail in this application.

[0123] It can be understood that step E2 can be inputting the target speech feature into a convolutional neural network, extracting local feature information in the target speech feature through the convolutional neural network, thereby obtaining the intermediate speech feature. Specifically, the target speech feature can be extracted locally through a CNN convolutional neural network, and the output of the CNN convolutional neural network is determined as the intermediate speech feature. Also, step E3 can be inputting the intermediate speech feature into a time cycle network module, extracting the time dependency information in the intermediate speech feature through the time cycle network module, thereby obtaining the speech emotion feature. Specifically, the time dependency information in the intermediate speech feature can be captured through a long short-term memory network (LSTM), and the output of the LSTM network is determined as the speech emotion feature.

[0124] Step 170: jointly train the initialized emotion recognition model according to the stage emotion features, the facial emotion features and the voice emotion features to obtain the trained emotion recognition model.

[0125] In an embodiment of the present application, based on the emotion recognition results corresponding to the stage emotion features, facial emotion features and voice emotion features, the target loss value corresponding to the emotion recognition results can be determined, and then the emotion recognition model can be jointly trained based on the target loss value to obtain a trained emotion recognition model.

[0126] The step 170, jointly training the initialized emotion recognition model according to the stage emotion feature, the face emotion feature and the voice emotion feature to obtain the trained emotion recognition model, comprises:

[0127] F1. According to the emotion features of the stage, feature fusion and classification are performed on the facial emotion features and the voice emotion features to obtain a target fusion classification vector;

[0128] F2. Update the parameters of the initialized emotion recognition model according to the target fusion classification vector to obtain the trained emotion recognition model.

[0129] In an embodiment of the present application, step F1 may first be to perform feature fusion on stage emotion features, facial emotion features and voice emotion features, and then input the fused features into a classifier composed of multiple fully connected layers and a Softmax activation function, thereby obtaining a target fused classification vector. The size of the target fused classification vector is (N, M), where N is the batch size input to the classifier, and M is the total number of emotion recognition categories.

[0130] It is understandable that step F2 may first determine the category corresponding to the maximum probability in the target fusion classification vector as the final emotion recognition result, and then measure the recognition accuracy of the emotion recognition model by the consistency of the emotion recognition result with the corresponding real emotion label. If the emotion recognition result is closer to the corresponding real situation label, the emotion recognition model is more accurate in emotion recognition and the reliability is better; conversely, if the emotion recognition result is closer to the corresponding real situation label, the emotion recognition model is less accurate in emotion recognition and the reliability is worse. Therefore, in an embodiment of the present application, a target loss value may be determined based on the emotion recognition result and the corresponding real emotion label. The embodiment of the present application does not limit the loss function used to determine the target loss value. For example, the loss function may be a 0-1 loss function. Based on the target loss value obtained through training, the back propagation algorithm may be used to update the parameters of the model, and a trained emotion recognition model may be obtained after several rounds of iteration. The specific number of iterations may be preset, or the training may be considered completed when the test set meets the accuracy requirements.

[0131] The following describes in detail an emotion recognition system for ship drivers proposed according to an embodiment of the present application with reference to the accompanying drawings.

[0132] Reference Figure 3 , an emotion recognition system for ship drivers proposed in an embodiment of the present application includes:

[0133] The first processing unit 101 is used to obtain the face data, voice data and staged emotional state of the ship driver to be emotion recognized;

[0134] The second processing unit 102 is used to input the face data, the voice data and the staged emotional state into the trained emotion recognition model to obtain the emotion recognition result of the ship driver;

[0135] The emotion recognition model includes a parallel voice network framework, a face network framework and a stage emotion network framework, and the trained emotion recognition model is obtained by the following steps:

[0136] Obtain the target ship driver's stage emotion training data, face raw data and voice raw data;

[0137] Performing a first preprocessing on the original face data to obtain face training data, and performing a second preprocessing on the original voice data to obtain voice training data;

[0138] Inputting the stage emotion training data into the stage emotion network framework for feature extraction to obtain stage emotion features, wherein the stage emotion features are used to characterize the weights of multiple emotion types corresponding to the target ship driver;

[0139] Inputting the face training data into the face network framework to extract features and obtain face emotion features, and inputting the voice training data into the voice network framework to extract features and obtain voice emotion features;

[0140] The initialized emotion recognition model is jointly trained according to the stage emotion features, the facial emotion features and the voice emotion features to obtain the trained emotion recognition model.

[0141] Reference Figure 4 , the embodiment of the present application further provides an electronic device, including:

[0142] at least one processor 201;

[0143] At least one memory 202, used to store at least one program;

[0144] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the above method embodiment.

[0145] Similarly, it can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0146] The embodiment of the present application further provides a computer-readable storage medium, in which a program executable by the processor 201 is stored. The program executable by the processor 201 is used to implement the above-mentioned method embodiment when executed by the processor 201.

[0147] Similarly, the contents of the above method embodiments are all applicable to the computer-readable storage medium embodiments. The functions specifically implemented by the computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0148] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the application is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part described as a larger operation is performed independently.

[0149] In addition, although the present application is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features can be integrated into a single physical device and / or software module, or one or more functions and / or features can be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present application. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional techniques of the engineer. Therefore, those skilled in the art can implement the present application set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present application, which is determined by the full scope of the attached claims and their equivalents.

[0150] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method of the embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0151] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0152] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), a fiber optic device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0153] It should be understood that the various parts of the present application can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0154] In the above description of this specification, the description with reference to the terms "one embodiment / example", "another embodiment / example" or "certain embodiments / examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0155] Although the embodiments of the present application have been shown and described, those skilled in the art will appreciate that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present application, and that the scope of the present application is defined by the claims and their equivalents.

[0156] The above is a specific description of the preferred implementation of the present application, but the present application is not limited to the embodiments. Technical personnel familiar with the field can make various equivalent modifications or substitutions without violating the spirit of the present application. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present application.

Claims

1. A method for identifying emotions of ship drivers, characterized in that: include: Obtain the facial data, voice data and staged emotional state of the ship driver to be identified; Inputting the face data, the voice data and the staged emotional state into a trained emotion recognition model to obtain an emotion recognition result of the ship driver; The emotion recognition model includes a parallel voice network framework, a face network framework and a stage emotion network framework, and the trained emotion recognition model is obtained by the following steps: Obtain the target ship driver's stage emotion training data, face raw data and voice raw data; Performing a first preprocessing on the original face data to obtain face training data, and performing a second preprocessing on the original voice data to obtain voice training data; Inputting the stage emotion training data into the stage emotion network framework for feature extraction to obtain stage emotion features, wherein the stage emotion features are used to characterize the weights of multiple emotion types corresponding to the target ship driver; Inputting the face training data into the face network framework to extract features and obtain face emotion features, and inputting the voice training data into the voice network framework to extract features and obtain voice emotion features; The initialized emotion recognition model is jointly trained according to the stage emotion features, the facial emotion features and the voice emotion features to obtain the trained emotion recognition model.

2. The emotion recognition method according to claim 1, characterized in that: The first preprocessing of the original face data to obtain face training data includes: Get a random occlusion rectangle and brightness level; Cleaning and editing the original face data to obtain intermediate face data; Based on the occlusion rectangle and the brightness level, data enhancement is performed on the intermediate face data to obtain the face training data; The second preprocessing of the original speech data to obtain speech training data includes: Performing voice data cleaning on the original voice data to obtain voice intermediate data; The speech intermediate data is clipped to obtain the speech training data.

3. The emotion recognition method according to claim 1, characterized in that: The stage emotion network framework includes a weighted network module and a transformation network module. The stage emotion training data is input into the stage emotion network framework for feature extraction to obtain stage emotion features, including: Inputting the stage emotion training data into the weighted network module for weighted summation to obtain intermediate emotion features; The intermediate emotion features are input into the transformation network module for nonlinear transformation to obtain the stage emotion features.

4. The emotion recognition method according to claim 1, characterized in that: The step of inputting the face training data into the face network framework to extract features and obtain facial emotion features includes: Extracting temporal attention features from the face training data to obtain temporal features of the face; Performing spatial attention feature extraction on the temporal features of the human face to obtain temporal and spatial features of the human face; The face spatiotemporal features are sampled at multiple time scales to obtain the face emotion features.

5. The emotion recognition method according to claim 4, characterized in that: The step of performing multi-time scale upsampling on the face spatiotemporal features to obtain the face emotion features includes: Performing short-time-scale sampling on the face spatiotemporal features to obtain short-time-scale face features; Performing mesoscale sampling on the face spatiotemporal features to obtain mesoscale face features; Performing long-time-scale sampling on the face spatiotemporal features to obtain long-time-scale face features; According to the short-time scale facial features and the medium-time scale facial features, channel splicing is performed on the long-time scale facial features to obtain the facial emotion features; Among them, the short-time scale facial features, the medium-time scale facial features and the long-time scale facial features are aligned with each other in the time dimension.

6. The emotion recognition method according to claim 1, characterized in that: The speech network framework includes a convolutional network module and a time recurrent network module. The speech training data is input into the speech network framework for feature extraction to obtain speech emotion features, including: Determining target speech features based on the speech training data; Inputting the target speech feature into the convolutional network module to extract local features and obtain intermediate speech features; The intermediate speech features are input into the time recurrent network module to extract time-dependent features to obtain the speech emotion features.

7. The emotion recognition method according to claim 1, characterized in that: The step of jointly training the initialized emotion recognition model according to the stage emotion features, the facial emotion features and the voice emotion features to obtain the trained emotion recognition model comprises: According to the stage emotion features, feature fusion and classification are performed on the facial emotion features and the voice emotion features to obtain a target fusion classification vector; According to the target fusion classification vector, the parameters of the initialized emotion recognition model are updated to obtain the trained emotion recognition model.

8. An emotion recognition system for ship drivers, characterized in that: include: The first processing unit is used to obtain the face data, voice data and staged emotional state of the ship driver to be emotion recognized; A second processing unit is used to input the face data, the voice data and the staged emotional state into the trained emotion recognition model to obtain an emotion recognition result of the ship driver; The emotion recognition model includes a parallel voice network framework, a face network framework and a stage emotion network framework, and the trained emotion recognition model is obtained by the following steps: Obtain the target ship driver's stage emotion training data, face raw data and voice raw data; Performing a first preprocessing on the original face data to obtain face training data, and performing a second preprocessing on the original voice data to obtain voice training data; Inputting the stage emotion training data into the stage emotion network framework for feature extraction to obtain stage emotion features, wherein the stage emotion features are used to characterize the weights of multiple emotion types corresponding to the target ship driver; Inputting the face training data into the face network framework to extract features and obtain face emotion features, and inputting the voice training data into the voice network framework to extract features and obtain voice emotion features; The initialized emotion recognition model is jointly trained according to the stage emotion features, the facial emotion features and the voice emotion features to obtain the trained emotion recognition model.

9. A computer device, characterized in that: include: at least one processor; at least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a program executable by a processor, characterized in that: The program executable by the processor is used to implement the method according to any one of claims 1 to 7 when executed by the processor.

Images