Intelligent cabin emotion interaction method and system and storage medium

Abstract

The invention discloses an intelligent cabin emotion interaction method and system and a storage medium. The method comprises the steps that multi-modal feature data and current environment feature data of a user are collected in real time; carrying out fusion processing on the multi-modal feature data and the current environment feature data based on a preset feature fusion algorithm, and obtaining an emotion recognition result through an emotion database based on a fusion result; performing sentiment analysis by adopting a sentiment interaction model based on the sentiment recognition result to obtain a current interaction strategy; and generating a cooperative control instruction in cooperation with the intelligent cabin system based on the current interaction strategy so as to control the intelligent cabin system to execute a corresponding interaction operation. According to the method provided by the invention, the defects of user emotion recognition and lack of adaptive interaction strategies of an intelligent cockpit system in the prior art are effectively overcome, and the adaptive interaction strategies are selected by accurately recognizing the user emotion, so that the user emotion is effectively adjusted, and personalized interaction experience is provided.

Description

Technical Field

[0001] This application belongs to the field of automotive electronics technology, specifically relating to a smart cockpit emotional interaction method, system, and storage medium. Background Technology

[0002] With the continuous development of automotive intelligence, the role of in-vehicle cockpit systems in the driving and riding process is becoming increasingly important. However, most existing cockpit systems only focus on providing basic navigation and entertainment functions, lacking the ability to perceive and respond to the user's emotional state. During driving or riding, users may experience different emotions for various reasons, such as fatigue, anxiety, and excitement. Existing driver monitoring systems mainly focus on fatigue and distracted driving, and have not yet delved into a comprehensive level of emotion recognition. Although the importance of affective computing has been recognized, the application of current technology in real-world driving environments still faces challenges. Single-modal emotion recognition technology is easily affected by external interference in real-world driving environments, resulting in insufficient accuracy and stability. Most existing intelligent cockpit systems adopt standardized interactive feedback, lacking the ability to dynamically adjust interaction strategies based on the user's emotional state. Although some systems have begun to integrate modules such as haptic feedback and voice acquisition, the synergy between these modules has not been fully utilized, and the interaction methods lack emotional adaptability. Summary of the Invention

[0003] To address the aforementioned technical issues, this application proposes an intelligent cockpit emotional interaction method, system, and storage medium capable of real-time monitoring of the user's emotional state and adaptively generating interaction strategies.

[0004] Specifically, this application proposes an intelligent cockpit emotional interaction method, including: Real-time collection of user multimodal feature data and current environmental feature data.

[0005] The multimodal feature data and the current environment feature data are fused based on a preset feature fusion algorithm, and the emotion recognition result is obtained through an emotion database based on the fusion result.

[0006] Based on the emotion recognition results, an emotion interaction model is used to perform emotion analysis in order to obtain the current interaction strategy.

[0007] Furthermore, based on the current interaction strategy, the intelligent cockpit system generates collaborative control commands to control the intelligent cockpit system to perform corresponding interactive operations.

[0008] In the above technical solution, by fusing multimodal feature data and current environmental feature data, the system can significantly improve the accuracy of emotion recognition in complex driving environments and effectively cope with changes in ambient lighting and noise interference. Based on the emotion recognition results, an emotion interaction model is used to analyze emotions and obtain interaction strategies, making human-computer interaction more natural and smooth, significantly improving the user experience. The system can automatically adjust the interaction method and feedback intensity according to the user's emotional state, reducing interaction friction and lowering the user's cognitive burden. Through the established emotion database and emotion interaction model, the system can adapt to the emotional expression habits of different users, improving the user experience for different users.

[0009] As one implementation method, before the real-time acquisition of the user's multimodal feature data and the current environment feature data, the method further includes: An emotion database is constructed based on historical multimodal feature data and historical environmental feature data, and the emotion recognition results corresponding to various historical multimodal feature data and historical environmental feature data are written into the emotion database; an emotion interaction model is constructed based on historical emotion recognition results and historical interaction strategies.

[0010] By constructing a sentiment database using historical multimodal feature data and historical environmental feature data, and performing correlation analysis on historical data, potential sentiment patterns can be uncovered, improving the system's adaptability to complex scenarios. Long-term accumulated user historical data can be used to train personalized sentiment models, effectively reducing the false positive rate of sentiment recognition. Sentiment interaction models can be constructed using historical sentiment recognition results and historical interaction strategies. These historical interaction strategies enable reinforcement learning of the sentiment interaction model, optimizing the strategy generation logic.

[0011] Furthermore, the process of fusing the multimodal feature data and the current environment feature data based on a preset feature fusion algorithm includes: Based on the current environmental feature data, the weights of each modality feature data are adjusted in real time, and the adjusted modality feature data are fused to obtain the current sentiment features.

[0012] Dynamic weight adjustments can significantly improve the environmental perception and emotion recognition accuracy of intelligent cockpit emotion interaction systems. By reducing the weight of unreliable modalities and increasing the weight of reliable modalities in real time based on environmental feature data—for example, reducing the weight of sound and increasing the weight of user facial expressions in noisy environments—feature distortion caused by weighting can be effectively avoided. By adjusting the weights of feature data from different modalities, the complementary information between multimodal feature data can be highlighted, ensuring the accuracy of the acquired emotion features.

[0013] Furthermore, obtaining the emotion recognition result based on the emotion database includes: The current emotional features are matched with the emotional database to obtain emotional recognition results; the emotional recognition results are written into the emotional database based on a preset privacy protection strategy.

[0014] By matching current sentiment features with historical data in the sentiment database, the accuracy of sentiment recognition is improved. If the current sentiment features cannot match existing sentiment recognition results in the database, new labels can be added after cluster analysis, thereby enriching the sentiment database. The preset privacy protection strategy stores only abstract sentiment features, thus avoiding the leakage of raw sensor data and effectively protecting user privacy.

[0015] Furthermore, the step of employing an emotional interaction model for sentiment analysis to obtain the current interaction strategy includes: If the emotion recognition result is a negative emotion, then the emotion interaction model is used to generate an emotion mitigation interaction strategy; if the emotion recognition result is a positive emotion, then the emotion interaction model is used to generate an emotion enhancement interaction strategy.

[0016] By employing differentiated interaction methods for different emotional states, adaptive interaction strategies can make the generated interaction strategies more reliable. Generating emotion-mitigation interaction strategies can effectively alleviate negative emotions in users, thereby improving driving stability and safety. When users are in a positive emotional state, emotion-enhancing interaction strategies can enhance their positive experience and create a more pleasant driving environment.

[0017] Furthermore, the generation of collaborative control commands by the collaborative intelligent cockpit system based on the current interaction strategy includes: The current interaction strategy is integrated with the intelligent cockpit system to generate collaborative control commands; the collaborative control commands are transmitted to the corresponding interaction module of the intelligent cockpit system to control the interaction module to perform the corresponding interactive operation.

[0018] By integrating current interaction strategies with intelligent cockpit systems, cross-system emotional perception sharing and collaborative response can be achieved. For example, heterogeneous devices such as air conditioning, seats, in-vehicle screens, and audio systems can respond collaboratively, thereby effectively influencing the user's emotional state through multi-faceted interactive operations.

[0019] Furthermore, after controlling the intelligent cockpit system to perform the corresponding interactive operations, it also includes: The system monitors the performance status of the intelligent cockpit system in real time and obtains user feedback information, and optimizes the emotional interaction model based on the performance status of the intelligent cockpit system and / or the feedback information.

[0020] By monitoring the cockpit system's performance in real time and acquiring user feedback, the intelligent cockpit emotional interaction system possesses the ability to continuously evolve. For example, if it detects that a user consistently lowers the volume after interacting with the system, it can automatically use vibration alerts instead of voice alerts the next time. Through continuous optimization of the emotional interaction model, it becomes increasingly mature, and the interactive actions taken in response to the user's emotional state become more accurate.

[0021] Based on the same inventive concept, this application also proposes a system for an intelligent cockpit emotional interaction method, the system comprising: The data acquisition module is used to collect multimodal feature data of users and current environmental feature data in real time.

[0022] The emotion recognition module is used to fuse the multimodal feature data and the current environment feature data based on a preset feature fusion algorithm, and obtain the emotion recognition result through a preset emotion database based on the fusion result.

[0023] The interaction analysis module is used to perform emotion analysis based on the emotion recognition results using a preset emotion interaction model in order to obtain the current interaction strategy.

[0024] Additionally, a collaborative control module is used to generate collaborative control commands based on the current interaction strategy in collaboration with the intelligent cockpit system, so as to control the intelligent cockpit system to perform corresponding interactive operations.

[0025] Furthermore, the data acquisition module includes a multimodal feature acquisition module and an environmental feature acquisition module.

[0026] The multimodal feature acquisition module is used to collect the user's multimodal feature data in real time; the environmental feature acquisition module is used to collect the current environmental feature data in real time.

[0027] The emotion recognition module includes a feature fusion module and an emotion matching module.

[0028] The feature fusion module is used to adjust the weights of each modality feature data in real time based on the current environmental feature data, and then fuse the adjusted modality feature data to obtain the current emotional features; the emotion matching module is used to match the current emotional features with the emotion database to obtain the emotion recognition result.

[0029] Based on the same inventive concept, this application also proposes a computer-readable storage medium storing computer-executable instructions, which, when executed by a control processor, implement the intelligent cockpit emotional interaction method.

[0030] Compared with the prior art, this application has at least the following beneficial effects: The intelligent cockpit emotion interaction method proposed in this application effectively solves the technical problem that current intelligent cockpit systems lack the ability to dynamically adjust interaction strategies based on the user's emotional state. By fusing multimodal feature data and current environmental feature data, the system can significantly improve the accuracy of emotion recognition in complex driving environments and effectively cope with changes in ambient lighting and noise interference. Based on the emotion recognition results, an emotion interaction model is used to perform emotion analysis and obtain interaction strategies, making human-computer interaction more natural and smooth, significantly improving the user experience. The system can automatically adjust the interaction method and feedback intensity according to the user's emotional state, reducing interaction friction and lowering the user's cognitive burden. Through the established emotion database and emotion interaction model, the system can adapt to the emotional expression habits of different users, improving the user experience for different users. Attached Figure Description

[0031] Figure 1 This is a flowchart illustrating the intelligent cockpit emotional interaction method in an embodiment of this application.

[0032] Figure 2 This is a flowchart illustrating the multimodal emotion feature fusion in an embodiment of this application.

[0033] Figure 3 This is a schematic diagram of an intelligent cockpit emotional interaction system shown in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0036] Example 1: Please refer to Figure 1 The intelligent cockpit emotional interaction method mainly includes steps S100 to S400.

[0037] Step S100 includes: real-time acquisition of the user's multimodal feature data and current environmental feature data. The multimodal feature data may include at least real-time acquired data such as the user's facial expressions, voice features, and physiological signals. The current environmental feature data may include at least data such as current vehicle speed, current weather, current temperature, current lighting conditions, and current ambient volume. Multiple sensors with different modalities can be used to acquire the user's multimodal feature data in real time, while vehicle-mounted sensors can be used to acquire the current environmental feature data in real time.

[0038] Step S200 includes: fusing the multimodal feature data and current environmental feature data based on a preset feature fusion algorithm, and obtaining emotion recognition results through an emotion database based on the fusion result. The preset feature fusion algorithm can primarily be an adaptive weighted multimodal emotion feature fusion algorithm, which dynamically adjusts the weights of each modality according to current environmental conditions to ensure optimal emotion recognition results in different environments. For example, the weight of facial expression data is reduced in low-light conditions, and the weight of speech feature modality data is reduced in noisy environments. The emotion database mainly records historical multimodal feature data and historical environmental feature data of users, as well as the corresponding emotion recognition results. Through continuous observation and recording of users' emotion recognition results, a personalized emotion expression model is constructed to adapt to the emotion expression habits of different users.

[0039] Step S300 includes: performing emotion analysis using an emotion interaction model based on the emotion recognition results to obtain the current interaction strategy. The emotion interaction model can primarily be a multi-dimensional emotion interaction strategy matrix, which employs differentiated interaction methods for different emotional states of the user. For example, when negative emotions are detected in the driver, the driver's negative emotions can be alleviated by adjusting the cabin environment, providing information support, or suggesting a rest. When positive emotions are detected in the driver, the user's positive experience can be enhanced through multi-modal feedback such as touch, sound, and light, creating a more pleasant driving environment.

[0040] Furthermore, step S400 includes: generating collaborative control commands based on the current interaction strategy in collaboration with the intelligent cockpit system to control the intelligent cockpit system to perform corresponding interactive operations. The intelligent cockpit system may primarily include ADAS (Advanced Driver Assistance Systems), a navigation system, an infotainment system, a seat adjustment system, etc.

[0041] In practice, multimodal sensors, including cameras, sound sensors, and non-contact sensors, are used to collect real-time data on the driver's facial expressions, voice, and physiological signals. Onboard sensors collect real-time data on the current environment, such as weather, temperature, and noise levels. An adaptive weighted multimodal emotion feature fusion algorithm integrates the collected multimodal and environmental data to obtain the driver's emotional characteristics. Based on these characteristics, an emotion recognition result is obtained from a pre-built emotion database using extensive historical data. An emotion interaction model is used to analyze this result, generating corresponding interaction strategies. These strategies are then transmitted to the intelligent cockpit system, which generates collaborative control commands. These commands control the intelligent cockpit system to perform corresponding interactive operations to effectively regulate the driver's emotional state.

[0042] In some embodiments, please refer to Figure 2 This is a flowchart of multimodal emotion feature fusion. Facial features can be dynamically recognized using deep learning algorithms, and HPFS (High Purity Feature Separation) technology can reduce the impact of lighting and posture changes on recognition accuracy. Voice features can be extracted from speech signals using improved acoustic feature extraction techniques, including prosody, tone quality, and spectral characteristics. This is combined with a driver voice emotion recognition network, GL-DSED (Global-Local Dual-Stream Emotion Decoder), which integrates global acoustic features and local spectral features, to achieve stable voice emotion recognition in noisy driving environments. Microphone array technology can be used to achieve specific action-specific sound pickup and actively eliminate various noises. Physiological signal detection can be achieved by monitoring the driver's heart rate variability, respiratory rate, and skin conductance using non-contact sensors. In conjunction with in-vehicle cameras, information on facial blood flow changes can be collected, enabling imperceptible physiological signal acquisition. In contextual awareness, information such as vehicle driving status, environmental conditions, and driving task difficulty can be collected to provide background support for user emotion recognition, improving the environmental adaptability of emotion recognition. An adaptive weighted multimodal emotion feature fusion algorithm can dynamically adjust the weights of each modality based on the current environmental conditions, ensuring superior emotion recognition performance in various environments. For example, the weight of the visual modality can be reduced in low-light conditions, and the weight of the speech modality can be reduced in noisy environments.

[0043] Preferably, before the real-time acquisition of the user's multimodal feature data and the current environment feature data, the method further includes: An emotion database is constructed based on historical multimodal feature data and historical environmental feature data, and the emotion recognition results corresponding to various historical multimodal feature data and historical environmental feature data are written into the emotion database; an emotion interaction model is constructed based on historical emotion recognition results and historical interaction strategies.

[0044] A sentiment database, constructed using historical multimodal and environmental feature data, records users' emotional expression patterns, enabling the creation of personalized sentiment expression models to adapt to the emotional expression habits of different users. By recording users' emotional responses in different contexts, it provides a continuous understanding of emotions across driving conversations.

[0045] Preferably, the process of fusing the multimodal feature data and the current environment feature data based on a preset feature fusion algorithm includes: Based on the current environmental feature data, the weights of each modality feature data are adjusted in real time, and the adjusted modality feature data are fused to obtain the current sentiment features.

[0046] For example, in high-intensity light environments, the weight of visual modality feature data can be reduced in real time, and in noisy environments, the weight of speech feature data can be reduced.

[0047] Preferably, obtaining the emotion recognition result based on the emotion database includes: The current emotional features are matched with the emotional database to obtain emotional recognition results; the emotional recognition results are written into the emotional database based on a preset privacy protection strategy.

[0048] The system matches the user's personalized emotional features recorded in the emotional database with the current emotional features. If a match is found, the corresponding emotional recognition result is output. Otherwise, a new emotional recognition result is created based on the current emotional features and the new emotional recognition result is recorded.

[0049] Preferably, the step of using an emotional interaction model to perform sentiment analysis to obtain the current interaction strategy includes: If the emotion recognition result is a negative emotion, then the emotion interaction model is used to generate an emotion mitigation interaction strategy; if the emotion recognition result is a positive emotion, then the emotion interaction model is used to generate an emotion enhancement interaction strategy.

[0050] The emotion recognition results are analyzed using the aforementioned emotion interaction model, allowing for differentiated interaction methods based on different emotional states. For example, if the emotion recognition result is negative, an emotion-relieving interaction strategy can be employed, such as adjusting the cabin environment or playing music to alleviate negative emotions. When the emotion recognition result is positive, emotion-enhancing interaction strategies can be used, such as increasing music volume or controlling ambient lighting, to enhance the user's positive experience and create a more pleasant driving environment. Personalized interactive services can also be provided based on the user's emotion recognition results. For instance, playing favorite or suitable music when the user is feeling down, or simplifying the display screen interface to reduce the user's cognitive burden when the user is anxious.

[0051] Preferably, the step of generating collaborative control commands based on the current interaction strategy in the collaborative intelligent cockpit system includes: The current interaction strategy is integrated with the intelligent cockpit system to generate collaborative control commands; the collaborative control commands are transmitted to the corresponding interaction module of the intelligent cockpit system to control the interaction module to perform the corresponding interactive operation.

[0052] For example, the interaction strategy can be deeply integrated with the vehicle's ADAS, navigation system, infotainment system, seat adjustment system, etc., to generate corresponding collaborative control commands, thereby realizing cross-system emotional perception sharing and collaborative response.

[0053] Preferably, after controlling the intelligent cockpit system to perform the corresponding interactive operation, the system further includes: The system monitors the performance status of the intelligent cockpit system in real time and obtains user feedback information, and optimizes the emotional interaction model based on the performance status of the intelligent cockpit system and / or the feedback information.

[0054] For example, if the user's initial interaction is to increase the volume, and the user subsequently lowers the volume multiple times, the volume-increasing operation will be removed from subsequent interactions in the same scenario, and the interaction strategy will be updated in real time to the emotion interaction model. The accuracy of emotion recognition and interaction strategies is continuously optimized by monitoring the performance of the intelligent cockpit system in real time and obtaining user feedback.

[0055] Example 2: Please refer to Figure 3 This application also proposes a system based on the intelligent cockpit emotional interaction method described in Embodiment 1. The system mainly includes: a data acquisition module, an emotion recognition module, an interaction analysis module, and a collaborative control module.

[0056] The data acquisition module is used to collect multimodal feature data of the user and current environmental feature data in real time. This module employs multimodal sensors to collect various feature data of the user in real time, such as facial feature data, voice feature data, and physiological signal feature data of the driver. It also uses onboard sensors to collect information such as the vehicle's driving status and the current environmental status of the vehicle.

[0057] The emotion recognition module is used to fuse the multimodal feature data and the current environment feature data based on a preset feature fusion algorithm, and obtain the emotion recognition result through a preset emotion database based on the fusion result. The preset feature fusion algorithm can be an adaptive weighted multimodal emotion feature fusion algorithm, which dynamically adjusts the weights of each modality feature data in the multimodal feature data according to the current environment feature data, ensuring accurate emotion recognition results in different environments.

[0058] The interaction analysis module is used to perform emotion analysis based on the emotion recognition results using a preset emotion interaction model to obtain the current interaction strategy. The emotion interaction model can adopt differentiated interaction strategies according to different emotion recognition results. For example, when the driver's emotional state is negative, an emotion mitigation interaction strategy can be used to alleviate the user's negative emotions; when the user's emotional state is positive, an emotion enhancement interaction strategy can be used to respond to the user's positive emotions and provide a more pleasant driving environment.

[0059] In addition, a collaborative control module is used to generate collaborative control commands based on the current interaction strategy in conjunction with the intelligent cockpit system, so as to control the intelligent cockpit system to perform corresponding interactive operations. For example, by coordinating with the display screen system, intelligent ambient lighting system, intelligent air conditioning, intelligent seats, and other systems based on the current interaction strategy, the generated collaborative control commands can control the air conditioning, seats, ambient lighting, music, and display content in the cockpit.

[0060] Preferably, the data acquisition module includes a multimodal feature acquisition module and an environmental feature acquisition module.

[0061] The multimodal feature acquisition module is used to collect the user's multimodal feature data in real time. Multimodal sensors can be used to collect this data; for example, a camera can capture facial data, and deep learning algorithms can be used to recognize facial expressions and extract facial feature data. Microphones can be used to collect the user's voice data, employing microphone array technology for zoned sound pickup at specific seats and actively eliminating various noises. Acoustic feature extraction technology can be used to extract prosody, tone quality, and spectrum features from the collected speech signals. Non-contact sensors can be used to detect the driver's physiological signal characteristics.

[0062] The environmental feature acquisition module is used to collect current environmental feature data in real time. This primarily involves using onboard sensors to record the vehicle's driving status, the weather conditions, and environmental features such as sound and light.

[0063] The emotion recognition module includes a feature fusion module and an emotion matching module.

[0064] The feature fusion module is used to adjust the weights of each modality feature data in real time based on the current environmental feature data, and then fuse the adjusted modality feature data to obtain the current emotional features. It mainly employs an adaptive feature fusion algorithm, for example, reducing the weight of visual modality feature data in low light or high light conditions, and reducing the weight of speech modality feature data in noisy environments.

[0065] The emotion matching module is used to match the current emotion features with the emotion database to obtain emotion recognition results. It matches the current features with historical emotion features in the emotion database to obtain corresponding emotion recognition results. Furthermore, it continuously optimizes the emotion database by updating it in real time based on the latest obtained emotion recognition results, effectively improving the accuracy of the emotion recognition results obtained through emotion matching.

[0066] Example 3: This application also proposes a computer-readable storage medium, the computer-readable storage medium comprising: The computer-readable storage medium stores computer-executable instructions; When the computer-executable instructions are executed by the control processor, the intelligent cockpit emotional interaction method described in Embodiment 1 is implemented.

[0067] The computer-readable storage medium can be implemented entirely or partially by software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented entirely or partially as a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in the computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)).

[0068] In summary, the intelligent cockpit emotion interaction method proposed in this application effectively solves the technical problem of current intelligent cockpit systems lacking the ability to dynamically adjust interaction strategies based on user emotional states. By fusing multimodal feature data and current environmental feature data, the system significantly improves the accuracy of emotion recognition in complex driving environments and effectively copes with changes in ambient lighting and noise interference. Based on the emotion recognition results, an emotion interaction model is used to perform emotion analysis and obtain interaction strategies, making human-computer interaction more natural and fluid, significantly improving the user experience. The system can automatically adjust the interaction method and feedback intensity according to the user's emotional state, reducing interaction friction and lowering the user's cognitive burden. Through the established emotion database and emotion interaction model, the system can adapt to the emotional expression habits of different users, improving the user experience for different users.

[0069] In the several embodiments provided in this application, it will be understood that each block in the flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the figures. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved.

[0070] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause an electronic device to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0071] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of this application. It should be understood that the above descriptions are merely specific embodiments of this application and are not intended to limit the scope of protection of this application. In particular, it should be noted that any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application for those skilled in the art.

Claims

1. A smart cockpit emotional interaction method, characterized in that, The intelligent cockpit emotional interaction method includes: Real-time collection of user multimodal feature data and current environmental feature data; The multimodal feature data and the current environment feature data are fused based on a preset feature fusion algorithm, and the emotion recognition result is obtained through an emotion database based on the fusion result. Based on the emotion recognition results, an emotion interaction model is used to perform emotion analysis in order to obtain the current interaction strategy. Furthermore, based on the current interaction strategy, the intelligent cockpit system generates collaborative control commands to control the intelligent cockpit system to perform corresponding interactive operations.

2. The intelligent cockpit emotional interaction method according to claim 1, characterized in that, Before the real-time acquisition of user multimodal feature data and current environmental feature data, the following steps are also included: A sentiment database is constructed based on historical multimodal feature data and historical environmental feature data, and the sentiment recognition results corresponding to various historical multimodal feature data and historical environmental feature data are written into the sentiment database. An emotion interaction model is constructed based on historical emotion recognition results and historical interaction strategies.

3. The intelligent cockpit emotional interaction method according to claim 2, characterized in that, The process of fusing the multimodal feature data and the current environment feature data based on a preset feature fusion algorithm includes: Based on the current environmental feature data, the weights of each modality feature data are adjusted in real time, and the adjusted modality feature data are fused to obtain the current sentiment features.

4. The intelligent cockpit emotional interaction method according to claim 3, characterized in that, The process of obtaining emotion recognition results based on an emotion database includes: The current emotion features are matched against the emotion database to obtain emotion recognition results; The emotion recognition results are written into the emotion database based on a preset privacy protection strategy.

5. The intelligent cockpit emotional interaction method according to claim 1, characterized in that, The step of using an emotional interaction model for sentiment analysis to obtain the current interaction strategy includes: If the emotion recognition result is a negative emotion, then the emotion interaction model is used to generate an emotion mitigation interaction strategy; If the emotion recognition result is a positive emotion, then the emotion interaction model is used to generate an emotion-enhancing interaction strategy.

6. The intelligent cockpit emotional interaction method according to claim 1, characterized in that, The generation of collaborative control commands by the collaborative intelligent cockpit system based on the current interaction strategy includes: The current interaction strategy is integrated with the intelligent cockpit system to generate collaborative control commands; The collaborative control command is transmitted to the corresponding interaction module of the intelligent cockpit system to control the interaction module to perform the corresponding interactive operation.

7. The intelligent cockpit emotional interaction method according to claim 1, characterized in that, After controlling the intelligent cockpit system to perform the corresponding interactive operations, it also includes: The system monitors the performance status of the intelligent cockpit system in real time and obtains user feedback information, and optimizes the emotional interaction model based on the performance status of the intelligent cockpit system and / or the feedback information.

8. A system based on the intelligent cockpit emotional interaction method according to any one of claims 1-7, characterized in that, The system includes: The data acquisition module is used to collect multimodal feature data of users and current environmental feature data in real time; The emotion recognition module is used to fuse the multimodal feature data and the current environment feature data based on a preset feature fusion algorithm, and to obtain the emotion recognition result through a preset emotion database based on the fusion result; The interaction analysis module is used to perform emotion analysis based on the emotion recognition results using a preset emotion interaction model in order to obtain the current interaction strategy; Additionally, a collaborative control module is used to generate collaborative control commands based on the current interaction strategy in collaboration with the intelligent cockpit system, so as to control the intelligent cockpit system to perform corresponding interactive operations.

9. The system of the intelligent cockpit emotional interaction method according to claim 8, characterized in that, The data acquisition module includes a multimodal feature acquisition module and an environmental feature acquisition module; The multimodal feature acquisition module is used to acquire the user's multimodal feature data in real time. The environmental feature acquisition module is used to collect current environmental feature data in real time; The emotion recognition module includes a feature fusion module and an emotion matching module; The feature fusion module is used to adjust the weights of each modality feature data in real time based on the current environmental feature data, and to fuse the adjusted modality feature data to obtain the current sentiment features. The emotion matching module is used to match the current emotion features with the emotion database to obtain the emotion recognition result.

10. A computer-readable storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed by the control processor, the intelligent cockpit emotional interaction method as described in any one of claims 1-7 is implemented.

Images