Driver multi-modal emotion recognition method based on semantic communication

Abstract

The invention relates to a driver multi-mode emotion recognition method based on semantic communication, and belongs to the technical field of Internet of Vehicles, and the method comprises the following steps: S1, collecting the visual, audio and text data of a driver, and carrying out the spatial-temporal feature extraction; s2, extracting emotional common sense relation characteristics of the image and the text; s3, carrying out semantic coding and channel coding; s4, performing channel decoding and semantic decoding; and S5, carrying out multi-modal feature fusion and emotion classification. According to the invention, efficient and reliable transmission of multi-mode emotion features in a low signal-to-noise ratio environment is realized, and the real-time processing capability of a driver emotion recognition system under an end-side cloud architecture is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle networking and relates to a driver multimodal emotion recognition method based on semantic communication. Background Art

[0002] With the continuous development of intelligent connected vehicles, vehicle intelligence has become an important trend in the development of future vehicles. Since drivers are easily affected by negative emotions, they induce dangerous driving behaviors, which has a serious impact on road safety. Therefore, establishing an accurate and real-time driver emotion recognition system can effectively ensure the driving safety of drivers and prevent road traffic accidents caused by drivers. At present, the existing driver emotion recognition methods mainly use facial data and physiological data, and there is little research on multimodal data with driving context characteristics, and there is no good consideration on how to fully integrate multimodal data at the feature fusion layer to improve the performance of emotion recognition methods. On the other hand, with the continuous increase in in-vehicle intelligent applications, the local computing resources available to the vehicle itself are becoming more and more limited. In the Internet of Vehicles environment, the transmission data between the vehicle and the edge layer continues to increase, and the bandwidth resources that a single vehicle can have are becoming less and less. Therefore, how to build an accurate and real-time driver emotion recognition system in the actual Internet of Vehicles environment is of great significance to the safe driving of future intelligent connected vehicles.

[0003] At present, the research on emotion recognition mainly focuses on single-modal data, and emotion recognition based on EEG signals is one of the hottest research areas. Nakisa et al. reviewed the most advanced EEG-based features to classify different emotions in "Evolutionary computation algorithms for feature selection of EEG-based emotion recognition using mobile sensors" [in Expert Syst. Appl., 2018, 93: 143-155]. However, existing studies have shown that in complex situations, a single modality may provide poor robustness, and the emotion recognition performance of multimodal data fusion is better than that of a single modality. Therefore, multimodal fusion technology has received increasing attention. Ghaleb et al. in "Multimodal fusion based on information gain for emotion recognition in the wild" [in Proc. Intell. Syst. Conf., 2017, 814-823] used a multimodal fusion framework to fuse video and audio data for emotion recognition, achieving better results than ever before. Tavakoli et al. proposed a framework called HARMONY in “HARMONY: A human-centered multimodal driving study in the wild” [in IEEE Access, 2021, 9: 23956-23978] and explored multimodal longitudinal driving studies in real driving scenarios. Existing attention mechanisms are often used in fusion layers to learn adaptive fusion weights of different modalities. Mou et al. proposed an attention-based multimodal framework CNN-LSTM-Attention to detect driver stress and drowsiness in “Isotropic self-supervised learning for driver drowsiness detection with attention-based multimodal fusion” [in IEEE Trans. Multimedia, 2021, 25: 529-542], combined with a self-supervised learning module. Although these methods have achieved good results, they do not take into account the interaction between different modalities. On the other hand, with the continuous development of intelligent connected vehicles, in order to achieve better intelligent control, a vehicle network data transmission model with a three-layer architecture of end-edge-cloud has been proposed.As the number of vehicles continues to increase, the effective bandwidth of a single roadside unit device to provide data transmission for each vehicle begins to decrease. How to accurately transmit data under low signal-to-noise ratio is a major challenge facing the current architecture. For data transmission in low signal-to-noise ratio environments, Xie et al. proposed a semantic communication system based on deep learning in "Deep learning enabled semantic communication systems" [in IEEE Transactions on Signal Processing, 2021, 69: 2663-2675], which achieved better transmission performance indicators of text under low signal-to-noise ratio than traditional communication systems. Yang et al. implemented a multimodal semantic transmission system in "Emotion-Aware Scene Adaptation: A Bandwidth-Efficient Approach for Generating Animated Shorts" [in Sensors, 2024, 24 (5): 1660], which converted vision into text feature expressions through a semantic encoder, providing a solution for quickly generating a wide range of media forms and quantities for user intentions. Although existing semantic communication has good performance in transmitting text semantic features, there are few methods for extracting and transmitting semantic features of audio and video modalities in the Internet of Vehicles, and the parallel transmission of multimodal semantic features has not been achieved.

[0004] Although a lot of research results have been achieved in the current multimodal emotion recognition technology, in the actual Internet of Vehicles environment, the existing multimodal emotion recognition models still find it difficult to effectively mine the emotional information of different modalities, and do not fully consider the interaction when different modal features are fused, especially the effective fusion of temporal or spatial dependent modal features. On the other hand, the previous lightweight driver emotion recognition model can be directly deployed to the vehicle terminal with its low model complexity, but for the current multimodal emotion recognition model with large volume and computational complexity, split deployment is required. Based on the edge-cloud architecture in the Internet of Vehicles, the feature extraction part with lower complexity is deployed downward to the vehicle terminal, and the modal fusion reasoning part with higher complexity is deployed upward to the cloud server, while the roadside unit collects and transmits data between the terminal and the cloud. However, due to the communication between the roadside unit and multiple vehicles, the wireless channel bandwidth resources between a vehicle and its corresponding roadside unit are very limited. Therefore, in the actual Internet of Vehicles environment with low signal-to-noise ratio, the transmission efficiency and accuracy of data between the vehicle terminal and the roadside unit are low.

[0005] In the current Internet of Vehicles environment, the existing multimodal emotion recognition methods for drivers have problems in model recognition accuracy and real-time data transmission. On the one hand, the existing multimodal emotion recognition models still find it difficult to effectively mine the emotional information of different modalities, and do not fully consider the interaction when different modal features are fused, especially the effective fusion of temporal or spatial dependent modal features. On the other hand, with the continuous development of intelligent connected vehicles, the number of in-vehicle intelligent applications continues to increase, making the in-vehicle local system undertake more and more computing tasks. Under the premise of ensuring the computing resources required for critical vehicle tasks, the computing resources used for non-critical computing tasks are increasingly limited. Therefore, the large model deployment adopts the end-edge cloud architecture for hierarchical deployment and processing. As the number of vehicle terminals in the coverage area of ​​the vehicle roadside unit increases, the bandwidth resources that each vehicle can have in the local area covered by the roadside unit are very limited, making it difficult to achieve efficient transmission of multimodal data in a low signal-to-noise ratio environment, resulting in a large delay in model recognition. Summary of the invention

[0006] In view of this, an object of the present invention is to provide a driver multimodal emotion recognition method based on semantic communication.

[0007] In order to achieve the above object, the present invention provides the following technical solutions:

[0008] A driver multimodal emotion recognition method based on semantic communication includes the following steps:

[0009] S1: Collect the driver’s visual, audio and text data and extract spatiotemporal features;

[0010] S2: Extract emotional common sense relationship features of images and texts;

[0011] S3: semantic coding and channel coding;

[0012] S4: channel decoding and semantic decoding;

[0013] S5: Multimodal feature fusion and emotion classification.

[0014] Furthermore, the step S1 of collecting the driver's visual, audio and text data and extracting spatiotemporal features specifically includes the following steps:

[0015] S11: Input trimodal data {X m |m∈{v,a,t}}where m represents one of the three modalities, v represents the visual modality, a represents the audio modality, and t represents the text modality; the trimodal data X m Input the ConvLSTM module respectively to obtain the first layer features of the three modes:

[0016]

[0017] Where l∈[1,N] represents the level of the ConvLSTM module, N represents the total number of layers of the ConvLSTM module, d represents the d-dimensional convolutional LSTM operation, ConvLSTM represents the ConvLSTM network operation, BatchNorm, Dropout and AvgPolling represent batch normalization, Dropout and average pooling operations respectively. and Represent the output results of ConvLSTM network and batch normalization operation respectively, The output representation of the l-th layer ConvLSTM module of modality m. Here, d = 2 indicates that a two-dimensional convolutional LSTM operation is used to capture spatiotemporal features;

[0018] S12: Through the N-layer ConvLSTM module operation, the feature representation {H} of the three modalities {v, a, t} of vision, audio and text is finally obtained v ,H a ,H t}, the number of ConvLSTM module layers is N=3.

[0019] Furthermore, the extraction of emotion common sense relationship features of the image and text in step S2 specifically includes the following steps:

[0020] The COMET model is used to continuously extract the relationship features between facial vision and text modalities, and obtain the common sense emotional relationship between the two modalities of image and text. and

[0021]

[0022]

[0023] Among them, CLIP(·) means obtaining text description information of the image through the CLIP model, and COMET(·) means extracting common sense relationship features of the modality through the COMET model; X t V and X t Respectively represent the image text description and original text modality data obtained through the CLIP model; and Represent the emotional common sense relationship features of images and texts respectively.

[0024] Further, the semantic coding and channel coding in step S3 includes the following steps:

[0025] S31: Input the three modalities and two common sense relationship features into the semantic encoder to obtain the semantic features of each modality;

[0026]

[0027] Among them, CrossAtten(A,B) means obtaining the Q matrix through A, obtaining the K and V matrices through B, and then calculating the cross attention, SelfAtten(A) means calculating the self-attention of A; K v and Y v Represents visual emotion common sense features and visual emotion semantic features respectively; S v and S a Represent visual and audio semantic features respectively, S t Calculation method and S v same;

[0028] S32: The semantic features of the three modalities are concatenated and encoded through a channel encoder composed of fully connected layers:

[0029]

[0030] Z=N·(FC encoder (S))+η

[0031] Among them, S represents the three-modal fusion semantic features, FC encoder (·) represents the fully connected operation of the channel encoder, N represents the random complex channel coefficient, η~N(0,δ 2 ) means the mean is 0 and the variance is δ 2 Gaussian white noise, Z represents the transmitted signal.

[0032] Further, the channel decoding and semantic decoding in step S4 specifically include the following steps:

[0033] S41: The received signal is passed through a channel decoder and the three modal semantic features are recovered through a Split operation:

[0034]

[0035] in, represents the recovered channel semantic features, FC decoder (·) represents the full connection operation of the channel decoder, Split(·) represents the split operation, and Respectively represent the restored visual, audio and text semantic features;

[0036] S42: Input the semantic features of the three modalities into the semantic decoder to obtain the original modal features of each modality restored after semantic transmission:

[0037]

[0038] Among them, BiLSTM(·) represents the bidirectional long short-term memory network operation; and They represent the restored visual emotion semantic features, fine-grained emotion features, and common sense weighted semantic features, respectively. v , β BiLSTM and β CrossAtten Respectively represent the corresponding weights; and Represent the restored visual and audio features respectively, Calculation method and Similarly, in order to simplify annotation, subsequent feature fusion uses It is represented by {H v ,H a ,H t}.

[0039] Furthermore, the multimodal feature fusion and emotion classification in step S5 specifically includes the following steps:

[0040] S51: Through the mixed attention fusion operation, for each specific modality, it is used as the dominant modality to perform cross-attention and self-attention operations with the remaining other modalities; for the facial expression modality, its cross-attention is based on the facial visual feature H v As the dominant feature, respectively a and H t To perform cross fusion:

[0041] [Q v ,K v ,V v ]=[H v W v Q ,H v W v K ,H v W v V ]

[0042] [Q v ,K a ,V a ]=[H v W v Q ,H a W a K ,H a W a V ]

[0043] [Q v ,K t ,V t]=[H v W v Q ,H t W t K ,H t W t V ]

[0044]

[0045] Among them, Q v The Q matrix obtained by representing the dominant visual features of the face, K v , K a and K t The K matrices obtained by representing visual, audio and text features, V v 、V a and V t Represents the V matrix obtained by audio and text features respectively; W v Q Represented by visual features H v Calculate the weight matrix of the Q matrix. The other W matrices have similar meanings. CA(·) and Softmax(·) represent the cross attention and Softmax calculation operations respectively. Represents the self-attention feature of facial vision, and Respectively represent the dominant facial visual features H v With audio feature H a and text features H t Cross-attention features obtained by cross-attention fusion;

[0046] The same operation is used to process audio and text modalities, and the self-attention and cross-attention features of the three modalities are obtained;

[0047] S52: Combine the features of each modality to obtain the cross-fusion features of the three modalities:

[0048]

[0049] in, represents the splicing operation along the feature direction, H V , H A and H T Respectively represent the cross-fusion features of facial vision, audio and text splicing; H∈R n×m represents the trimodal features after the three cross-fusion features are concatenated, n represents the number of time layers represented by the three modal features, m represents the number of features of the three modalities, and H r,c represents the eigenvalue of the rth row and cth column;

[0050] S53: The correlation between features is calculated, and the self-attention mechanism is used to weight different features. Finally, different features are weighted and spliced ​​to obtain the final weighted fusion feature H. α :

[0051] k t =tanh(w·H t +b)

[0052]

[0053] Among them, H t =(H t,1 ,H t,2 ,...,H t,n ) T , (t∈[1,m]) represents the t-th column vector of the trimodal feature, kt represents the activation result of the t-th column vector after the tanh function operation, α t represents the weighted value of the tth column vector, H α ∈R n×1 Represents the weighted fusion feature obtained by weighted addition of each feature;

[0054] S54: The final emotion category is obtained through two fully connected layers and the Softmax function; the loss function is the standard cross entropy loss Among them, y i represents the discrimination probability of different emotion categories of the i-th sample, N is the number of samples in the video dataset, and CrossEntropy is the result of the cross entropy loss function.

[0055] The beneficial effect of the present invention is that the present invention designs a driver multimodal emotion recognition method based on semantic communication, aiming to realize the rapid and accurate recognition of driver emotions in a real vehicle network deployment environment. First, a driver multimodal emotion recognition architecture with end-edge-cloud collaboration is proposed. Based on the end-edge-cloud architecture in the vehicle network, the feature extraction part with lower complexity is deployed to the vehicle terminal, the modal fusion reasoning part with higher complexity is deployed to the cloud server, and the roadside unit is used for data collection, distribution and transmission between the terminal and the cloud. The three-layer architecture is used to share the computing load, which greatly reduces the large amount of calculations required by the vehicle locally, thereby realizing the deployment and application of complex models for emotion recognition. Secondly, a multimodal spatiotemporal feature extraction and hybrid attention fusion method is designed. The ConvLSTM network is used to effectively extract the temporal dependency and deep feature information of different modalities, and the hierarchical features with time correlation are obtained, thereby realizing the full extraction of modal features. On the other hand, the self-attention and cross-attention mechanisms are used to mix and fuse the multimodal features, and the weighted splicing of different dominant modal features is achieved to achieve the full fusion of different modal emotional information, which provides a methodological basis for accurately identifying the driver's driving emotions. Finally, a multimodal emotion recognition semantic communication system was constructed. A multimodal emotion semantic shared knowledge base was introduced in the terminal feature extraction layer and the edge semantic parsing layer, and external public common sense was used to reduce part of the public semantic transmission. The emotion-semantic codec and channel codec were used to jointly realize the semantic feature extraction and reception of multimodal emotion data, and the reconstruction of the final multimodal emotion features, so as to achieve efficient and reliable transmission of multimodal emotion features in a low signal-to-noise ratio environment, thereby improving the real-time processing capability of the driver emotion recognition system under the end-edge-cloud architecture.

[0056] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0058] Figure 1 This is a block diagram of the end-edge-cloud collaborative driver emotion recognition semantic communication system of the present invention;

[0059] Figure 2 The time-domain multi-level multi-modal hybrid attention emotion recognition model proposed by the present invention;

[0060] Figure 3This is the multimodal emotion recognition semantic communication system proposed by the present invention. DETAILED DESCRIPTION

[0061] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0062] It should be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present invention, and thus the drawings only show components related to the present invention rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0063] In the following description, numerous details are discussed to provide a more thorough explanation of the embodiments of the present invention. However, it is obvious to those skilled in the art that the embodiments of the present invention can be implemented without these specific details. In other embodiments, well-known structures and devices are shown in the form of block diagrams rather than in detail to avoid making the embodiments of the present invention difficult to understand.

[0064] 1. Multimodal data analysis of connected vehicles with device-edge-cloud collaboration

[0065] Figure 1The end-edge-cloud collaborative driver emotion recognition semantic communication system provided by the specific embodiment of the present invention. Since the driver multimodal emotion recognition model is large in size and relatively complex in calculation, it is different from the small-volume emotion recognition model that can be directly deployed in the vehicle in the past. The model needs to be split and deployed, that is, the feature extraction module with smaller volume and calculation amount is sunk, and the fusion classification module with larger parameter amount and calculation amount is floated. Therefore, in order to reduce the local computing burden of the vehicle, an end-edge-cloud collaborative architecture is designed, which deploys the feature extraction module to the vehicle and the fusion reasoning module to the cloud. By realizing the load balancing of the calculation, the local computing amount is effectively reduced and the model recognition efficiency is improved. First, the multimodal data of the vehicle terminal is input into the feature extractor and the COMET model respectively to obtain multimodal features and text common sense features. Then, the common semantic features and trimodal features are input into the emotion-semantic encoding module to obtain multimodal and common sense semantic features. Next, the two features are spliced, input into the channel encoding module for encoding, and sent to the physical channel for transmission. At the receiving end, the roadside unit of the edge layer decodes the received data through the channel decoding module to recover the multimodal and common sense semantic features. Then, the two semantic features are input into the emotion-semantic decoder, and the reconstructed multimodal emotion features are obtained by using the common sense semantic features, combined with the multimodal semantic features and the shared semantic information of the decoder. By constructing a multimodal emotion semantic communication system, efficient transmission of Internet of Vehicles data under low signal-to-noise ratio conditions is achieved, thereby improving the real-time processing capability of the driver emotion recognition system under the edge-cloud architecture.

[0066] 2. Driver Emotion Recognition

[0067] like Figure 2 The time-domain multi-level multi-modal mixed attention emotion recognition model is shown in the figure. The first is feature extraction, using stacked ConvLSTM modules to perform multi-modal time-domain and multi-level feature extraction to obtain sufficient emotion information for further semantic feature extraction. Secondly, the multimodal features are used to mine intra-modal and inter-modal information using self-attention and cross-attention, to achieve full fusion between modalities and obtain multimodal fusion features. Finally, the multimodal fusion features are input into the final emotion classification network to obtain the emotion recognition result. Specifically, the following steps are included:

[0068] (1) Multimodal feature extraction

[0069] The vehicle sensor collects the audio and video data of the driver during driving. The unified ConvLSTM module is used to extract features from the driver's text, audio and visual modal data {v, a, t}, thereby obtaining the corresponding features of different modalities. Among them, the ConvLSTM module consists of four parts: ConvLSTM network, batch normalization, Dropout and average pooling. First, the three-modal data is input into the ConvLSTM network to extract spatiotemporal features. Then, in order to speed up the model training process and prevent overfitting, batch normalization and Dropout operations are applied in sequence. Finally, the model parameters are further reduced through average pooling, resulting in a more lightweight model structure.

[0070] The input trimodal data is represented as: {X m |m∈{v,a,t}}. Where m represents one of the three modes. m Input the ConvLSTM module respectively to obtain the first layer features of the three modes:

[0071]

[0072] Where l∈[1,N] represents the level of the ConvLSTM module, N represents the total number of layers of the ConvLSTM module, d represents the d-dimensional convolutional LSTM operation, ConvLSTM represents the ConvLSTM network operation, BatchNorm, Dropout and AvgPolling represent batch normalization, Dropout and average pooling operations respectively. and Represent the output results of ConvLSTM network and batch normalization operation respectively, Represents the output representation of the l-th layer ConvLSTM module of modality m. Here, d = 2 indicates that a two-dimensional convolutional LSTM operation is used to capture spatiotemporal features.

[0073] Through the operation of N layers of ConvLSTM modules, the feature representation {H v ,H a ,H t}. The number of ConvLSTM module layers is N=3.

[0074] (2) Hybrid Attention Fusion

[0075] For each specific modality, it is used as the dominant modality to perform cross-attention and self-attention operations with the remaining modalities. Taking the facial expression modality as an example, for its cross-attention, the facial visual feature H v As the dominant feature, respectively a and Ht The specific steps are as follows: First, calculate the Q, K and V matrices in the cross attention respectively, and the dominant modal feature H v Calculate the Q matrix, H a and H t Calculate the K and V matrices. Then, perform dot product operations on the Q, K, and V matrices to obtain the similarity matrix. Finally, apply the Softmax function to each row of the similarity matrix to obtain the final cross attention matrix. For its self-attention operation, use the same dominant feature H as the Q matrix on the K and V matrices. v The calculation is done, and the other operations are the same as the cross attention. Finally, the self-attention and two cross-attention features under the facial expression dominant mode are obtained:

[0076] [Q v ,K v ,V v ]=[H v W v Q ,H v W v K ,H v W v V ]

[0077] [Q v ,K a ,V a ]=[H v W v Q ,H a W a K ,H a W a V ]

[0078] [Q v ,K t ,V t ]=[H v W v Q ,H t W t K ,H t W t V ]

[0079]

[0080] Among them, Q v The Q matrix obtained by representing the dominant visual features of the face, K v , K a and Kt The K matrices obtained by representing visual, audio and text features, V v 、V a and V t Represents the V matrix obtained by audio and text features respectively. v Q Represented by visual features H v Calculate the weight matrix of the Q matrix. The other W matrices have similar meanings. CA(·) and Softmax(·) represent the cross attention and Softmax calculation operations respectively. Represents the self-attention feature of facial vision, and Respectively represent the dominant facial visual features H v With audio feature H a and text features H t Cross-attention features obtained by cross-attention fusion.

[0081] The same operation is used to process audio and text modalities to obtain the self-attention and cross-attention features of the three modalities. The features of each modality are concatenated to obtain the cross-fusion features of the three modalities:

[0082]

[0083] in, represents the splicing operation along the feature direction, H V , H A and H T Represent the cross-fusion features of facial vision, audio and text splicing. n×m represents the trimodal features after the three cross-fusion features are concatenated, n represents the number of time layers represented by the three modal features, m represents the number of features of the three modalities, and H r,c represents the eigenvalue of the rth row and cth column.

[0084] By calculating the correlation between features, we can use the self-attention mechanism to weight different features, and finally weighted splice different features to obtain the final weighted fusion feature H α :

[0085] k t =tanh(w·H t +b)

[0086]

[0087] Among them, H t =(H t,1 ,H t,2 ,...,H t,n ) T,(t∈[1,m]) represents the tth column vector of the trimodal feature, k t Represents the activation result of the t-th column vector after the tanh function operation, α t represents the weighted value of the tth column vector, H α ∈R n×1 Represents the weighted fusion feature obtained by weighted addition of each feature.

[0088] (3) Classification and Identification

[0089] Through two layers of fully connected layers and the Softmax function, the final emotion category is obtained. The loss function is the standard cross entropy loss Among them, y i represents the probability of different emotion categories of the i-th sample. N is the number of samples in the video dataset, and CrossEntropy is the result of the cross entropy loss function.

[0090] 3. Multimodal semantic communication

[0091] like Figure 3 The multimodal emotion recognition semantic communication system is shown in the figure. First, the sending end performs semantic encoding and channel encoding on the multimodal data to obtain channel semantic features. Then, the receiving end reconstructs the multimodal features through the channel and semantic decoder.

[0092] Finally, the reconstructed multimodal features are sent to the cloud for modal fusion classification. The specific steps include:

[0093] (1) Sending end

[0094] First, the three-modal data of the driver are processed through a semantic encoder to obtain their respective semantic features. Then, the semantic features of the three modalities are concatenated. Finally, the concatenated features are input into a channel encoder to obtain channel semantic features, and the semantic signal is transmitted through a signal transmitter.

[0095] For the driver's three-modal data {v, a, t}, on the one hand, the feature representations of the three modes {H v ,H a ,H t On the other hand, the common sense relationship between vision and text is mined through the common sense knowledge base model COMET. By integrating the common sense relationship between emotion and modal features, the semantic extraction of multimodal features can be improved.

[0096] The COMET model can mine many different types of common sense relationships. For the task scenario of emotion recognition, five of the more important relationships are selected for extraction: effect xEffect, intention xIntent, reaction xReact, desire xWant, and need xNeed. By using the COMET model to continue to extract relationship features from facial vision and text modalities, we can obtain the common sense emotional relationships of the two modalities. and

[0097]

[0098] Among them, CLIP(·) means obtaining text description information of the image through the CLIP model, and COMET(·) means extracting common sense relationship features of the modality through the COMET model. t V and X t They respectively represent the image text description and original text modality data obtained through the CLIP model. and Represent the emotional common sense relationship features of images and texts respectively.

[0099] The three modalities and two common sense relationship features are input into the semantic encoder to obtain the semantic features of each modality. Specifically, for the visual and textual modalities, the modal features and their corresponding emotional common sense relationship features are input into their respective emotion-semantic encoders to obtain the visual and textual semantic features S v and S t ; For the audio modality, its features are input into the self-attention encoder to obtain the audio semantic features S a .

[0100]

[0101] S a =SelfAtten(H a )

[0102] Among them, CrossAtten(A,B) means obtaining the Q matrix through A, obtaining the K and V matrices through B, and then calculating the cross attention, and SelfAtten(A) means calculating the self-attention of A. K v and Y v Represents the common sense features of visual emotions and the semantic features of visual emotions respectively. v and S a Represent visual and audio semantic features respectively, S t Calculation method and S v same.

[0103] The semantic features of the three modalities are concatenated and encoded through a channel encoder composed of fully connected layers to obtain channel semantic features, which are finally sent out.

[0104]

[0105] Z=N·(FC encoder (S))+η

[0106] Among them, S represents the three-modal fusion semantic features, FC encoder (·) represents the fully connected operation of the channel encoder, N represents the random complex channel coefficient, η~N(0,δ 2 ) means the mean is 0 and the variance is δ 2 Gaussian white noise, Z represents the transmitted signal.

[0107] (2) Receiver

[0108] First, the signal receiver at the receiving end receives the channel semantic signal and restores the channel semantic features through the channel decoder. Then, the channel semantic features are split to obtain the semantic features before the three modal splicing. Finally, the semantic features of the three modalities are input into the semantic decoder to obtain the restored trimodal features.

[0109] The received signal is passed through the channel decoder and the three modal semantic features are recovered through the Split operation:

[0110]

[0111] in, represents the recovered channel semantic features, FC decoder (·) represents the full connection operation of the channel decoder, Split(·) represents the split operation, and Represent the restored visual, audio and text semantic features respectively.

[0112] The semantic features of the three modalities are input into the semantic decoder to obtain the original modal features restored after semantic transmission of each modality. Specifically, for the visual and textual modalities, the restored semantic features are input into their respective emotion-semantic decoders. The emotion-semantic encoder consists of three parts: the semantic features directly restored, the contextual fine-grained emotion information obtained through the BiLSTM model, and the common sense weighted semantic information obtained through cross-attention. The three are weightedly added and input into the self-attention network to obtain the final restored features of the visual and textual modalities. and For the audio modality, its features are input into the self-attention decoder to obtain the restored features of the audio

[0113]

[0114] Among them, BiLSTM(·) represents a bidirectional long short-term memory network operation. and represent the restored visual emotion semantic features, fine-grained emotion features, and common sense weighted semantic features, respectively, while β v , β BiLSTM and β CrossAtten Here, we choose β v =0.2,β BiLSTM =0.3,β CrossAtten =1-β v -β BiLSTM =0.5. and Represent the restored visual and audio features respectively, Calculation method and same.

[0115] In the above embodiments, the description's reference to "this embodiment" indicates that a particular feature, structure, or characteristic described in conjunction with the embodiment is included in at least some embodiments, but not necessarily all embodiments. Multiple occurrences of "this embodiment" do not necessarily all refer to the same embodiment.

[0116] In the above-described embodiments, although the invention has been described in conjunction with specific embodiments of the invention, many substitutions, modifications, and variations of these embodiments will be apparent to those of ordinary skill in the art based on the foregoing description. For example, other storage structures (e.g., dynamic RAM (DRAM)) may use the embodiments discussed. Embodiments of the invention are intended to encompass all such substitutions, modifications, and variations that fall within the broad scope of the appended claims.

[0117] This embodiment further provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, any one of the methods in this embodiment is implemented.

[0118] This embodiment also provides an electronic terminal, including: a processor and a memory;

[0119] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal executes any one of the methods in this embodiment.

[0120] The computer-readable storage medium in this embodiment can be understood by ordinary technicians in this field: all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0121] The electronic terminal provided in this embodiment includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected to the processor and the transceiver and complete communication with each other. The memory is used to store computer programs, the communication interface is used to communicate, and the processor and the transceiver are used to run computer programs so that the electronic terminal executes each step of the above method.

[0122] In this embodiment, the memory may include a random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk memory.

[0123] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0124] The present invention can be used in many general or special computing system environments or configurations, such as personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like.

[0125] The present invention may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present invention may also be practiced in distributed computing environments where tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0126] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A driver multimodal emotion recognition method based on semantic communication, characterized by: The following steps are involved: S1: Collect the driver’s visual, audio and text data and extract spatiotemporal features; S2: Extract emotional common sense relationship features of images and texts; S3: semantic coding and channel coding; S4: channel decoding and semantic decoding; S5: Multimodal feature fusion and emotion classification.

2. The driver multimodal emotion recognition method based on semantic communication according to claim 1 is characterized by: Step S1 collects the driver's visual, audio and text data and extracts spatiotemporal features, specifically including the following steps: S11: Input trimodal data {X m |m∈{v,a,t}}where m represents one of the three modalities, v represents the visual modality, a represents the audio modality, and t represents the text modality; the trimodal data X m Input the ConvLSTM module respectively to obtain the first layer features of the three modes: Where l∈[1,N] represents the level of the ConvLSTM module, N represents the total number of layers of the ConvLSTM module, d represents the d-dimensional convolutional LSTM operation, ConvLSTM represents the ConvLSTM network operation, BatchNorm, Dropout and AvgPolling represent batch normalization, Dropout and average pooling operations respectively. and Represent the output results of ConvLSTM network and batch normalization operation respectively, Represents the output representation of the l-th layer ConvLSTM module of modality m; d = 2 means using a two-dimensional convolutional LSTM operation to capture spatiotemporal features; S12: Through the N-layer ConvLSTM module operation, the feature representation {H} of the three modalities {v, a, t} of vision, audio and text is finally obtained v ,H a ,H t }, the number of ConvLSTM module layers is N=3.

3. The driver multimodal emotion recognition method based on semantic communication according to claim 1 is characterized in that: Step S2 extracts the emotion common sense relationship features of the image and text, specifically including the following steps: The COMET model is used to continuously extract the relationship features between facial vision and text modalities, and obtain the common sense emotional relationship between the two modalities of image and text. and Among them, CLIP(·) means obtaining text description information of the image through the CLIP model, and COMET(·) means extracting common sense relationship features of the modality through the COMET model; and X t Respectively represent the image text description and original text modality data obtained through the CLIP model; and Represent the emotional common sense relationship features of images and texts respectively.

4. The driver multimodal emotion recognition method based on semantic communication according to claim 1 is characterized in that: The semantic coding and channel coding in step S3 includes the following steps: S31: Input the three modalities and two common sense relationship features into the semantic encoder to obtain the semantic features of each modality; S a =SelfAtten(H a ) Among them, CrossAtten(A,B) means obtaining the Q matrix through A, obtaining the K and V matrices through B, and then calculating the cross attention, SelfAtten(A) means calculating the self-attention of A; K v and Y v Represents visual emotion common sense features and visual emotion semantic features respectively; S v and S a Represent visual and audio semantic features respectively, S t Calculation method and S v same; S32: The semantic features of the three modalities are concatenated and encoded through a channel encoder composed of fully connected layers: Z=N·(FC encoder (S))+η Among them, S represents the three-modal fusion semantic features, FC encoder (·) represents the fully connected operation of the channel encoder, N represents the random complex channel coefficient, η~N(0,δ 2 ) means the mean is 0 and the variance is δ 2 Gaussian white noise, Z represents the transmitted signal.

5. The driver multimodal emotion recognition method based on semantic communication according to claim 1 is characterized in that: The channel decoding and semantic decoding in step S4 specifically include the following steps: S41: The received signal is passed through a channel decoder and the three modal semantic features are recovered through a Split operation: in, represents the recovered channel semantic features, FC decoder (·) represents the full connection operation of the channel decoder, Split(·) represents the split operation, and Respectively represent the restored visual, audio and text semantic features; S42: Input the semantic features of the three modalities into the semantic decoder to obtain the original modal features of each modality restored after semantic transmission: Among them, BiLSTM(·) represents the bidirectional long short-term memory network operation; and They represent the restored visual emotion semantic features, fine-grained emotion features, and common sense weighted semantic features, respectively. v , β BiLSTM and β CrossAtten Respectively represent the corresponding weights; and Represent the restored visual and audio features respectively, Calculation method and Similarly, in order to simplify annotation, subsequent feature fusion uses It is represented by {H v ,H a ,H t }.

6. The driver multimodal emotion recognition method based on semantic communication according to claim 1 is characterized by: The multimodal feature fusion and emotion classification described in step S5 specifically includes the following steps: S51: Through the mixed attention fusion operation, for each specific modality, it is used as the dominant modality to perform cross-attention and self-attention operations with the remaining other modalities; for the facial expression modality, its cross-attention is based on the facial visual feature H v are the dominant features, respectively a and H t To perform cross fusion: Among them, Q v The Q matrix obtained by representing the dominant visual features of the face, K v , K a and K t The K matrices obtained by representing visual, audio and text features, V v 、V a and V t Represents the V matrix obtained by audio and text features respectively; Represented by visual features H v Calculate the weight matrix of the Q matrix. The other W matrices have similar meanings. CA(·) and Softmax(·) represent the cross attention and Softmax calculation operations respectively. Represents the self-attention feature of facial vision, and Respectively represent the dominant facial visual features H v With audio feature H a and text features H t Cross-attention features obtained by cross-attention fusion; The same operation is used to process audio and text modalities, and the self-attention and cross-attention features of the three modalities are obtained; S52: Combine the features of each modality to obtain the cross-fusion features of the three modalities: in, represents the splicing operation along the feature direction, H V , H A and H T Respectively represent the cross-fusion features of facial vision, audio and text splicing; H∈R n×m represents the trimodal features after the three cross-fusion features are concatenated, n represents the number of time layers represented by the three modal features, m represents the number of features of the three modalities, and H r,c represents the eigenvalue of the rth row and cth column; S53: The correlation between features is calculated, and the self-attention mechanism is used to weight different features. Finally, different features are weighted and spliced ​​to obtain the final weighted fusion feature H. α : k t =tanh(w·H t +b) Among them, H t =(H t,1 ,H t,2 ,...,H t,n ) T , (t∈[1,m]) represents the t-th column vector of the trimodal feature, kt represents the activation result of the t-th column vector after the tanh function operation, α t represents the weighted value of the tth column vector, H α ∈R n×1 Represents the weighted fusion feature obtained by weighted addition of each feature; S54: The final emotion category is obtained through two fully connected layers and the Softmax function; the loss function is the standard cross entropy loss Among them, y i represents the discrimination probability of different emotion categories of the i-th sample, N is the number of samples in the video dataset, and CrossEntropy is the result of the cross entropy loss function.

Images