In-vehicle emotion detection method and device, electronic equipment and vehicle

Abstract

The invention provides an in-vehicle emotion detection method and device and electronic equipment. The in-vehicle emotion detection method comprises the steps of obtaining visual data, voice data and physiological data; splicing and fusing the visual data, the voice data and the physiological data after the redundant data is removed to obtain long connection fusion data; extracting a first emotional feature in the long connection fusion data, wherein the first emotional feature comprises a first visual feature, a first voice feature and a first physiological feature; obtaining weights respectively corresponding to the first visual feature, the first voice feature and the first physiological feature; processing the first emotional feature through weight to obtain a second emotional feature; and outputting an emotion detection result for the second emotion feature. According to the embodiment of the invention, through fusion and weight adjustment of the multi-modal data, the adaptability and robustness in a complex driving environment are enhanced, so that the driving safety and comfort are effectively improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of vehicle intelligent technology, and in particular to an in-vehicle emotion detection method, device, electronic equipment and vehicle. Background Art

[0002] The driver's emotional state directly affects driving behavior and decision-making. Emotional fluctuations may lead to distraction, slow response, and even dangerous driving behaviors such as speeding, sudden braking, fatigue driving, etc. By identifying the driver's emotions in real time, potential danger signals can be discovered in time, thereby effectively preventing traffic accidents and improving driving safety.

[0003] However, in the related technologies, most systems rely on single-modal data input for emotion detection. However, in complex driving environments, the detection accuracy and reliability of these systems are significantly reduced. The variability and noise interference of the driving environment, such as light changes, obstructions, background noise, etc., will have a serious impact on single-modal emotion detection systems. In addition, the driver's emotional state may be expressed in many ways, and it is difficult for a single-modal system to fully capture this information, resulting in insufficient accuracy in emotion detection. Summary of the invention

[0004] Embodiments of the present application provide an in-vehicle emotion detection method, device, system, electronic device, and computer-readable storage medium.

[0005] In a first aspect, an embodiment of the present application provides a method for detecting emotions in a vehicle, the method comprising:

[0006] Acquire visual data, voice data, and physiological data;

[0007] The visual data, speech data and physiological data after removing redundant data are concatenated and fused to obtain long-connection fusion data;

[0008] Extracting a first emotion feature from the long connection fusion data, where the first emotion feature includes a first visual feature, a first voice feature, and a first physiological feature;

[0009] Obtaining weights corresponding to the first visual feature, the first speech feature, and the first physiological feature respectively;

[0010] Processing the first emotion feature by using the weight to obtain a second emotion feature;

[0011] Outputting an emotion detection result for the second emotion feature.

[0012] Optionally, after the step of acquiring weights respectively corresponding to the first visual feature, the first voice feature, and the first physiological feature, the method includes:

[0013] Determining whether the weights corresponding to the respective features in the first emotion feature are obtained;

[0014] If not, each feature in the first emotion feature is used as a node, and a pre-set probability relationship between each node is obtained to construct a Bayesian network;

[0015] Probabilistic inference is performed through the Bayesian network to calculate the posterior probabilities of different features;

[0016] Dynamically adjust the feature fusion strategy according to the posterior probability;

[0017] Processing the first emotion feature according to the feature fusion strategy to obtain a second emotion feature;

[0018] Outputting an emotion detection result for the second emotion feature.

[0019] Optionally, outputting an emotion detection result for the second emotion feature includes:

[0020] The second emotion feature is input into a preset neural network, and at least one emotion detection result corresponding to the second emotion feature is output.

[0021] Optionally, the step of concatenating and fusing the visual data, the voice data, and the physiological data after removing redundant data to obtain the long-connection fusion data includes:

[0022] Removing redundant data from the visual data, voice data, and physiological data to obtain target visual data, target voice data, and target physiological data;

[0023] The target visual data, target voice data and target physiological data are concatenated and fused to obtain long-connection fusion data.

[0024] Optionally, the processing the first emotion feature by using the weight to obtain the second emotion feature includes:

[0025] The first emotion feature and the weight are input into a preset neural fusion network, and a second emotion feature at a topological level is output.

[0026] Optionally, after the step of outputting an emotion detection result for the second emotion feature, the method includes:

[0027] The vehicle interior environment is adjusted in conjunction with the pre-acquired vehicle exterior environment information and the emotion detection result.

[0028] Optionally, the acquiring of visual data, voice data and physiological data includes:

[0029] Acquire a facial image corresponding to a person inside the vehicle and extract visual data from the facial image; and, acquire a voice signal corresponding to the person inside the vehicle and extract voice data from the voice signal; and, acquire a physiological signal corresponding to the person inside the vehicle and extract physiological data from the physiological signal.

[0030] In a second aspect, an embodiment of the present application provides an in-vehicle emotion detection device, the device comprising:

[0031] An acquisition module, used to acquire visual data, voice data and physiological data;

[0032] A splicing module is used to splice and fuse the visual data, speech data and physiological data after removing redundant data to obtain long-connection fusion data;

[0033] An extraction module, used to extract a first emotion feature from the long connection fusion data, where the first emotion feature includes a first visual feature, a first voice feature, and a first physiological feature;

[0034] An acquisition module, used to acquire weights corresponding to the first visual feature, the first speech feature, and the first physiological feature respectively;

[0035] a processing module, configured to process the first emotion feature by using the weight to obtain a second emotion feature;

[0036] An output module is used to output an emotion detection result for the second emotion feature.

[0037] Optionally, the device further comprises:

[0038] A judgment module, used to judge whether the weights corresponding to the respective features in the first emotion feature are obtained;

[0039] A construction module, for, if not, using each feature in the first emotion feature as a node, and obtaining a pre-set probability relationship between each node to construct a Bayesian network;

[0040] An inference module, used to perform probability inference through the Bayesian network and calculate the posterior probabilities of different features;

[0041] An adjustment module, used for dynamically adjusting the feature fusion strategy according to the posterior probability;

[0042] A second processing module, configured to process the first emotion feature according to the feature fusion strategy to obtain a second emotion feature;

[0043] The second output module is used to output the emotion detection result according to the second emotion feature.

[0044] Optionally, the output module or the second output module includes:

[0045] The output submodule is used to input the second emotion feature into a preset neural network and output at least one emotion detection result corresponding to the second emotion feature.

[0046] Optionally, the splicing module includes:

[0047] A first splicing submodule is used to remove redundant data in the visual data, voice data and physiological data to obtain target visual data, target voice data and target physiological data;

[0048] The second splicing submodule is used to splice and fuse the target visual data, the target voice data and the target physiological data to obtain long-connected fused data.

[0049] Optionally, the processing module includes:

[0050] The processing submodule is used to input the first emotion feature and the weight into a preset neural fusion network, and output a second emotion feature at a topological level.

[0051] Optionally, the device further comprises:

[0052] The linkage adjustment module is used to linkage adjust the vehicle's internal environment according to the pre-acquired vehicle external environment information and the emotion detection result.

[0053] Optionally, the acquisition module includes:

[0054] The acquisition submodule is used to acquire a facial image corresponding to the person inside the vehicle and extract visual data from the facial image; and to acquire a voice signal corresponding to the person inside the vehicle and extract voice data from the voice signal; and to acquire a physiological signal corresponding to the person inside the vehicle and extract physiological data from the physiological signal.

[0055] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor, wherein the processor is configured to execute the instructions to implement any of the in-vehicle emotion detection methods described above.

[0056] In a fourth aspect, an embodiment of the present application further provides a vehicle, comprising the in-vehicle emotion detection device as described above.

[0057] In an embodiment of the present application, visual data, voice data and physiological data are obtained; the visual data, voice data and physiological data after removing redundant data are spliced ​​and fused to obtain long-connected fused data; the first emotional feature in the long-connected fused data is extracted, and the first emotional feature includes a first visual feature, a first voice feature and a first physiological feature; the weights corresponding to the first visual feature, the first voice feature and the first physiological feature are obtained; the first emotional feature is processed by the weight to obtain a second emotional feature; and the emotion detection result is output for the second emotional feature. That is, in an embodiment of the present application, visual data, voice data and physiological data are obtained, and the long-connected fused data is obtained after the redundancy is obtained, and then the first emotional feature is extracted, including the first visual feature, the first voice feature and the first physiological feature. The weights corresponding to each feature are obtained, and the first emotional feature is processed by using these weights to obtain the second emotional feature, and finally the emotion detection result is output. This process not only improves the accuracy and reliability of emotion detection, but also enhances the adaptability and robustness in complex driving environments through the fusion of multimodal data and weight adjustment, thereby effectively improving driving safety and comfort.

[0058] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present application. Also, the same reference symbols are used throughout the accompanying drawings to represent the same components. In the accompanying drawings:

[0060] Figure 1 is a flowchart of the steps of a method for in-vehicle emotion detection provided by an embodiment of the present application;

[0061] Figure 2 is a device block diagram of an in-vehicle emotion detection device provided by an embodiment of the present application;

[0062] Figure 3 It is a structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0063] The exemplary embodiments of the present application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present application are shown in the accompanying drawings, it should be understood that the present application can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present application and to fully convey the scope of the present application to those skilled in the art.

[0064] It should be noted that in the embodiment of the present application, the steering system is analyzed based on the safety goal of avoiding excessive steering assistance. It can be concluded that the sensors, controllers and actuators of the steering system may have failures that violate the safety goal of loss of steering assistance. Therefore, by real-time monitoring of these failures, when the steering system confirms a failure that violates the safety goal, it enters the corresponding safety state, switches from the safety state of excessive steering assistance to the redundant system to continue responding to steering assistance requests, and can light up the corresponding fault lights or fault prompts on the instrument panel.

[0065] Figure 1 is a flowchart of a method for detecting emotions in a vehicle provided by an embodiment of the present application. Figure 1 As shown, the method may include:

[0066] Step 101, obtaining visual data, voice data and physiological data;

[0067] Furthermore, the obtaining of visual data, voice data and physiological data includes:

[0068] Acquire a facial image corresponding to a person inside the vehicle and extract visual data from the facial image; and, acquire a voice signal corresponding to the person inside the vehicle and extract voice data from the voice signal; and, acquire a physiological signal corresponding to the person inside the vehicle and extract physiological data from the physiological signal.

[0069] It should be noted that in the embodiment of the present application, multimodal data collected inside the vehicle is obtained, rather than single sensor data.

[0070] Specifically, for visual data, the camera installed in the car can be used to capture the facial image, and the facial expression features can be extracted through ResNet. The residual structure of the ResNet network enables the model to effectively learn complex features in the core network. Combined with the 3D visual neural network (3D-CNN) to process continuous video frame data, the dynamic changes of facial expressions, such as the recognition of micro-expressions, can be captured. 3D-CNN has the ability to capture the spatiotemporal features of video data and can recognize the emotions at the time.

[0071] For voice data, the microphone collects the captured voice signal and uses LSTM to analyze the time series characteristics of the voice signal. LSTM network is good at processing long-term dependent sequence data and can identify emotional characteristics in voice. Combined with the multi-head attention mechanism, the emotional information in the voice signal is further analyzed. The multi-attention mechanism assigns different weights to different parts of the voice signal through different attention heads, thereby improving the accuracy of emotion recognition.

[0072] For physiological data, physiological sensors (such as heart rate and skin galvanic response sensors) are used to collect and capture physiological signals. These signals are processed through a proximity sensor network (TCN) to extract the target features. TCN can analyze physiological signal data in a shorter time frame and identify the current reaction of the vehicle. Gaussian model (GMM) mixture is used to simulate the statistical distribution of normal physiological signals and identify abnormal emotional states by comparing with real-time data.

[0073] Step 102, concatenating and fusing the visual data, speech data, and physiological data after removing redundant data to obtain long-connection fusion data;

[0074] Furthermore, the visual data, speech data and physiological data after removing redundant data are concatenated and fused to obtain long connection fusion data, including:

[0075] Removing redundant data from the visual data, voice data, and physiological data to obtain target visual data, target voice data, and target physiological data;

[0076] The target visual data, target voice data and target physiological data are concatenated and fused to obtain long-connection fusion data.

[0077] It should be noted that after collecting the multimodal data, redundant operations such as de-redundancy and redundancy can be performed on the collected data.

[0078] After the redundant operation, preliminary fusion is performed. Specifically, the features included in the data of different modalities are connected, that is, a simple fusion is first performed through a splicing (concatenation) operation. Feature splicing combines the data of each modality into a long connection so that the subsequent model can simultaneously consider features from different modalities. For example, in the embodiment of the present application, the visual feature support length is 256, the speech feature support length is 128, and the physiological signal feature support length is 64, then the feature support length after splicing is 448.

[0079] Step 103, extracting a first emotion feature from the long connection fusion data, where the first emotion feature includes a first visual feature, a first voice feature, and a first physiological feature;

[0080] It should be noted that in the embodiment of the present application, the spliced ​​features are input into a multi-layer collector (MLP) for nonlinear mapping and dimensionality reduction. MLP can extract higher gradient level emotion features by learning the complex relationship between different modalities.

[0081] Step 104, obtaining weights corresponding to the first visual feature, the first speech feature, and the first physiological feature respectively;

[0082] It should be noted that in the embodiment of the present application, a self-attention mechanism is applied to the features after preliminary estimation to further optimize feature fusion in a dynamic weighted manner. The self-attention mechanism can identify the most features and assign them higher weights by calculating the correlation between each element and other elements in the feature analysis (i.e., attention weight).

[0083] For example, when there are large changes in facial expressions, the weight of the visual feature will be increased, and when there are obvious changes in the voice signal, the weight of the voice feature will be higher. Therefore, the weights corresponding to the first visual feature, the first voice feature, and the first physiological feature will be obtained respectively.

[0084] In addition, in another embodiment, self-attention can be further extended to a multi-head attention mechanism, which calculates multiple different attention allocations through the attention mechanism to capture multiple relationships of features. This mechanism can focus on multiple important parts of features at the same time, thereby improving the accuracy of emotion detection.

[0085] Step 105, processing the first emotion feature by using the weight to obtain a second emotion feature;

[0086] Further, the processing the first emotion feature by the weight to obtain the second emotion feature includes:

[0087] The first emotion feature and the weight are input into a preset neural fusion network, and a second emotion feature at a topological level is output.

[0088] It should be noted that in the embodiment of the present application, the weight features of the self-mechanism attention output are input into a specially designed neural fusion network, namely, a preset neural fusion network. This preset neural fusion network can be a feature neural network (CNN) or a nonlinear network (RNN). The fused features are further processed according to the network to extract the emotion features at the topological level, namely, the second emotion features at the topological level.

[0089] Step 106: output an emotion detection result for the second emotion feature.

[0090] It should be noted that in an embodiment of the present application, the high-order emotion features processed by the fusion process are input into the final emotion prediction model. For example, the emotion prediction model may include a fully connected layer + Softmax classifier to generate the above-mentioned current emotion detection results, wherein the emotion detection results may include the occurrence probability distribution of multiple emotion categories. For example, the occurrence probability distribution of emotion categories such as anger, anxiety, and fatigue may be output.

[0091] Furthermore, in another embodiment, after step 104, the following steps may also be included:

[0092] Determining whether the weights corresponding to the respective features in the first emotion feature are obtained;

[0093] If not, each feature in the first emotion feature is used as a node, and a pre-set probability relationship between each node is obtained to construct a Bayesian network;

[0094] Probabilistic inference is performed through the Bayesian network to calculate the posterior probabilities of different features;

[0095] Dynamically adjust the feature fusion strategy according to the posterior probability;

[0096] Processing the first emotion feature according to the feature fusion strategy to obtain a second emotion feature;

[0097] Outputting an emotion detection result for the second emotion feature.

[0098] It should be noted that in an embodiment of the present application, after obtaining the weights corresponding to the first visual feature, the first speech feature and the first physiological feature respectively, when the weights are uncertain, that is, when there is still uncertainty in the weights after the self-attention mechanism, Bayesian fusion is adopted.

[0099] Specifically, first, features are extracted from different modalities. These features can be the result of processing by the self-attention mechanism, that is, the first emotional features. Secondly, the features of different modalities are used as nodes, and the conditional dependencies between them are defined. In this way, the probability relationship between the features of different modalities can be modeled, and the Bayesian network is used for probability inference to calculate the posterior probabilities of the features of different modalities. These posterior probabilities can be used to dynamically adjust the fusion strategy. Specifically, for example, according to the inferred posterior probabilities, the weights of the features of different modalities are dynamically adjusted. This dynamic adjustment can be made based on the current context information and historical data, so as to better cope with uncertainty. Finally, the decision output is made based on the fused features.

[0100] It should be noted that in the embodiment of the present application, the problem of weight uncertainty can be handled after the self-attention mechanism through the Bayesian fusion method. Bayesian fusion models the probabilistic relationship between different modal features through the Bayesian network, and dynamically adjusts the fusion strategy according to the context and historical data, so that it can still effectively deal with various uncertainties when facing complex driving bottlenecks.

[0101] It should be noted that, in the above two embodiments, both include the following contents:

[0102] That is, outputting the emotion detection result for the second emotion feature includes:

[0103] The second emotion feature is input into a preset neural network, and at least one emotion detection result corresponding to the second emotion feature is output.

[0104] That is, after the step of outputting the emotion detection result for the second emotion feature, the method includes:

[0105] The vehicle interior environment is adjusted in conjunction with the pre-acquired vehicle exterior environment information and the emotion detection result.

[0106] It should be noted that in the embodiments of the present application, the feedback mechanism can be continuously optimized by combining the emotion result prediction with the preset module (such as weather and road conditions) using the reinforcement learning algorithm, learning and adapting to the response mode of different feedbacks, and dynamically adjusting the reminder mode (such as audio prompts and vibrations). According to the user's emotional state and real-time environmental data, the in-car environment can be intelligently adjusted, such as automatically selecting music to play, adjusting the temperature or antenna settings, thereby effectively improving the user's emotional experience.

[0107] In an embodiment of the present application, visual data, voice data and physiological data are obtained; the visual data, voice data and physiological data after removing redundant data are spliced ​​and fused to obtain long-connected fused data; the first emotional feature in the long-connected fused data is extracted, and the first emotional feature includes a first visual feature, a first voice feature and a first physiological feature; the weights corresponding to the first visual feature, the first voice feature and the first physiological feature are obtained; the first emotional feature is processed by the weight to obtain a second emotional feature; and the emotion detection result is output for the second emotional feature. That is, in an embodiment of the present application, visual data, voice data and physiological data are obtained, and the long-connected fused data is obtained after the redundancy is removed, and then the first emotional feature is extracted, including the first visual feature, the first voice feature and the first physiological feature. The weights corresponding to each feature are obtained, and the first emotional feature is processed by using these weights to obtain the second emotional feature, and finally the emotion detection result is output. This process not only improves the accuracy and reliability of emotion detection, but also enhances the adaptability and robustness of the system in complex driving environments through the fusion of multimodal data and dynamic weight adjustment, thereby effectively improving driving safety and comfort.

[0108] In addition, the present application determines whether to obtain the weights corresponding to each feature in the first emotion fusion feature. If not, a Bayesian network is constructed and probability inference is performed to calculate the posterior probabilities of different features, and the feature fusion strategy is dynamically adjusted according to the posterior probabilities. This process not only improves the flexibility and adaptability of emotion detection, but also effectively copes with the uncertainty in complex driving environments through the probability inference of the Bayesian network, thereby significantly improving the accuracy and robustness of emotion detection and further enhancing driving safety and comfort.

[0109] Corresponding to the method provided in the embodiment of the in-vehicle emotion detection method of the present application, see Figure 2 The present application also provides a device block diagram of an in-vehicle emotion detection device. In this embodiment, the device includes:

[0110] An acquisition module, used to acquire visual data, voice data and physiological data;

[0111] A splicing module is used to splice and fuse the visual data, speech data and physiological data after removing redundant data to obtain long-connection fusion data;

[0112] An extraction module, used to extract a first emotion feature from the long connection fusion data, where the first emotion feature includes a first visual feature, a first voice feature, and a first physiological feature;

[0113] An acquisition module, used to acquire weights corresponding to the first visual feature, the first speech feature, and the first physiological feature respectively;

[0114] A processing module, configured to process the first emotion feature by using the weight to obtain a second emotion feature;

[0115] An output module is used to output an emotion detection result for the second emotion feature.

[0116] Optionally, the device further comprises:

[0117] A judgment module, used to judge whether the weights corresponding to the respective features in the first emotion feature are obtained;

[0118] A construction module, for, if not, using each feature in the first emotion feature as a node, and obtaining a pre-set probability relationship between each node to construct a Bayesian network;

[0119] An inference module, used to perform probability inference through the Bayesian network and calculate the posterior probabilities of different features;

[0120] An adjustment module, used for dynamically adjusting the feature fusion strategy according to the posterior probability;

[0121] A second processing module, configured to process the first emotion feature according to the feature fusion strategy to obtain a second emotion feature;

[0122] The second output module is used to output the emotion detection result according to the second emotion feature.

[0123] Optionally, the output module or the second output module includes:

[0124] The output submodule is used to input the second emotion feature into a preset neural network and output at least one emotion detection result corresponding to the second emotion feature.

[0125] Optionally, the splicing module includes:

[0126] A first splicing submodule is used to remove redundant data in the visual data, voice data and physiological data to obtain target visual data, target voice data and target physiological data;

[0127] The second splicing submodule is used to splice and fuse the target visual data, the target voice data and the target physiological data to obtain long-connected fused data.

[0128] Optionally, the processing module includes:

[0129] The processing submodule is used to input the first emotion feature and the weight into a preset neural fusion network, and output a second emotion feature at a topological level.

[0130] Optionally, the device further comprises:

[0131] The linkage adjustment module is used to linkage adjust the vehicle's internal environment according to the pre-acquired vehicle external environment information and the emotion detection result.

[0132] Optionally, the acquisition module includes:

[0133] The acquisition submodule is used to acquire a facial image corresponding to a person inside the vehicle and extract visual data from the facial image; and to acquire a voice signal corresponding to the person inside the vehicle and extract voice data from the voice signal; and to acquire a physiological signal corresponding to the person inside the vehicle and extract physiological data from the physiological signal.

[0134] In summary, an in-vehicle emotion detection device provided by an embodiment of the present application acquires visual data, voice data, and physiological data; splices and fuses the visual data, voice data, and physiological data after removing redundant data to obtain long-connected fusion data; extracts the first emotion feature in the long-connected fusion data, wherein the first emotion feature includes the first visual feature, the first voice feature, and the first physiological feature; acquires the weights corresponding to the first visual feature, the first voice feature, and the first physiological feature respectively; processes the first emotion feature by the weights to obtain the second emotion feature; and outputs the emotion detection result for the second emotion feature. That is, in an embodiment of the present application, by acquiring visual data, voice data, and physiological data, and splicing and fusing them after redundancy, long-connected fusion data is obtained, and then the first emotion feature is extracted, including the first visual feature, the first voice feature, and the first physiological feature. By acquiring the weights corresponding to each feature, and using these weights to process the first emotion feature, the second emotion feature is obtained, and finally the emotion detection result is output. This process not only improves the accuracy and reliability of emotion detection, but also enhances the adaptability and robustness of the system in complex driving environments through the fusion of multimodal data and dynamic weight adjustment, thereby effectively improving driving safety and comfort.

[0135] In addition, the present application determines whether to obtain the weights corresponding to each feature in the first emotion fusion feature. If not, a Bayesian network is constructed and probability inference is performed to calculate the posterior probabilities of different features, and the feature fusion strategy is dynamically adjusted according to the posterior probabilities. This process not only improves the flexibility and adaptability of emotion detection, but also effectively copes with the uncertainty in complex driving environments through the probability inference of the Bayesian network, thereby significantly improving the accuracy and robustness of emotion detection and further enhancing driving safety and comfort.

[0136] Figure 3 This is a structural diagram of an electronic device M00 provided in an embodiment of the present application. In the figure, the electronic device M00 includes a processor M01 and a memory M02. The electronic device includes: a processor; a memory for storing instructions executable by the processor, wherein the processor is configured to execute the instructions to implement any of the above-described in-vehicle emotion detection methods, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0137] In an embodiment of the present application, the memory M02 can be used to store software programs and various data. The memory M02 may mainly include a first storage area for storing programs or instructions and a second storage area for storing data, wherein the first storage area may store an operating system, an application program or instructions required for at least one function (such as a sound playback function, an image playback function, etc.), etc. In addition, the memory M02 may include a volatile memory or a non-volatile memory, or the memory x09 may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate synchronous dynamic random access memory (DDRSDRAM), an enhanced synchronous dynamic random access memory (ESDRAM), a synchronous link dynamic random access memory (SLDRAM) and a direct memory bus random access memory (DRRAM). The memory M02 in the embodiment of the present application includes but is not limited to these and any other suitable types of memories.

[0138] The processor M01 may include one or more processing units; optionally, the processor M01 integrates an application processor and a modem processor, wherein the application processor mainly processes operations related to an operating system, a user interface, and application programs, and the modem processor mainly processes wireless communication signals, such as a baseband processor. It is understandable that the modem processor may not be integrated into the processor M01.

[0139] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned in-vehicle emotion detection method embodiment are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0140] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer readable storage medium, such as a computer read-only memory ROM, a random access memory RAM, a magnetic disk or an optical disk.

[0141] An embodiment of the present application further provides a chip, which includes a processor and a communication interface, wherein the communication interface is coupled to the processor, and the processor is used to run programs or instructions to implement the various processes of the above-mentioned in-vehicle emotion detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0142] It should be understood that the chip involved in the embodiments of the present application can also be called a system-level chip, a system chip, a chip system or a system-on-chip chip, etc.

[0143] An embodiment of the present application also provides a storage medium. When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device can execute any of the in-vehicle emotion detection methods described above.

[0144] An embodiment of the present application provides a computer program product, which is stored in a storage medium. The program product is executed by at least one processor to implement the various processes of the above-mentioned in-vehicle emotion detection method embodiment, and can achieve the same technical effect. To avoid repetition, it will not be repeated here.

[0145] An embodiment of the present application also provides a vehicle, comprising the in-vehicle emotion detection device as described above.

[0146] It should be noted that, in this article, the terms "comprise", "include" or any other variant thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise one..." do not exclude the presence of other identical elements in the process, method, article or device including the element. In addition, it should be pointed out that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in reverse order according to the functions involved, for example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present application, or the part that contributes to the relevant technology, can be embodied in the form of a computer software product, which is stored in a storage medium (such as ROM / RAM, a disk, or an optical disk), and includes a number of instructions for a terminal (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0148] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present application, ordinary technicians in this field can also make many forms without departing from the purpose of the present application and the scope of protection of the claims, all of which are within the protection of the present application.

Claims

1. A method for detecting emotions in a vehicle, characterized in that: The method comprises: Acquire visual data, voice data, and physiological data; The visual data, speech data and physiological data after removing redundant data are concatenated and fused to obtain long-connection fusion data; Extracting a first emotion feature from the long connection fusion data, where the first emotion feature includes a first visual feature, a first voice feature, and a first physiological feature; Obtaining weights corresponding to the first visual feature, the first speech feature, and the first physiological feature respectively; Processing the first emotion feature by using the weight to obtain a second emotion feature; Outputting an emotion detection result for the second emotion feature.

2. The method according to claim 1, characterized in that After the step of acquiring weights corresponding to the first visual feature, the first voice feature, and the first physiological feature, the method includes: Determining whether the weights corresponding to the respective features in the first emotion feature are obtained; If not, each feature in the first emotion feature is used as a node, and a pre-set probability relationship between each node is obtained to construct a Bayesian network; Probabilistic inference is performed through the Bayesian network to calculate the posterior probabilities of different features; Dynamically adjust the feature fusion strategy according to the posterior probability; Processing the first emotion feature according to the feature fusion strategy to obtain a second emotion feature; Outputting an emotion detection result for the second emotion feature.

3. The method according to claim 1 or 2, characterized in that: Outputting the emotion detection result for the second emotion feature includes: The second emotion feature is input into a preset neural network, and an emotion detection result is output, where the emotion detection result includes a probability corresponding to at least one emotion category.

4. The method according to claim 1, characterized in that: The visual data, speech data and physiological data after removing redundant data are concatenated and fused to obtain long connection fusion data, including: Removing redundant data from the visual data, voice data, and physiological data to obtain target visual data, target voice data, and target physiological data; The target visual data, target voice data and target physiological data are concatenated and fused to obtain long-connection fusion data.

5. The method according to claim 1, characterized in that Processing the first emotion feature by using the weight to obtain the second emotion feature includes: The first emotion feature and the weight are input into a preset neural fusion network, and a second emotion feature at a topological level is output.

6. The method according to claim 1 or 2, characterized in that: After the step of outputting the emotion detection result for the second emotion feature, the method includes: The vehicle interior environment is adjusted in conjunction with the pre-acquired vehicle exterior environment information and the emotion detection result.

7. The method according to claim 1, characterized in that The acquisition of visual data, voice data and physiological data includes: Acquire a facial image corresponding to a person inside the vehicle and extract visual data from the facial image; and, acquire a voice signal corresponding to the person inside the vehicle and extract voice data from the voice signal; and, acquire a physiological signal corresponding to the person inside the vehicle and extract physiological data from the physiological signal.

8. An in-vehicle emotion detection device, characterized in that: The device comprises: A first acquisition module, used to acquire visual data, voice data and physiological data; A splicing module is used to splice and fuse the visual data, speech data and physiological data after removing redundant data to obtain long-connection fusion data; An extraction module, used to extract a first emotion feature from the long connection fusion data, where the first emotion feature includes a first visual feature, a first voice feature, and a first physiological feature; A second acquisition module, used to acquire weights corresponding to the first visual feature, the first speech feature, and the first physiological feature respectively; a processing module, configured to process the first emotion feature by using the weight to obtain a second emotion feature; An output module is used to output an emotion detection result for the second emotion feature.

9. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the in-vehicle emotion detection method as described in any one of claims 1 to 7.

10. A vehicle, characterized in that: The vehicle comprises the in-vehicle emotion detection device as claimed in any one of Claims 8.

Images