Cabin linkage control method, storage medium and vehicle

Abstract

The invention discloses a cabin linkage control method, a storage medium and a vehicle, which are applied to a vehicle-mounted control system, and the method comprises the following steps: collecting a face image of a person in a cabin, and determining a face interference state of the face image corresponding to the person; inputting the face image into a plurality of emotion detection models to obtain a corresponding output detection result; based on the face interference state, determining an evaluation weight corresponding to each emotion detection model, and performing weighted fusion on the detection result according to the evaluation weight to obtain the emotion state of the person; according to the emotional state, generating a cabin linkage strategy for transitioning the cabin from the current state to the target state; based on the cabin linkage strategy, at least one environment adjusting device in the cabin is controlled to perform progressive linkage action, and the detection results output by the plurality of emotion detection models are dynamically fused, so that the accuracy of emotion recognition in a complex in-vehicle environment is effectively improved, and emotion misrecognition of a single model under the condition of visual interference is avoided.

Description

Technical Field

[0001] This disclosure relates to the field of cockpit interaction technology, and in particular to a cockpit linkage control method, storage medium, and vehicle. Background Technology

[0002] With the continuous development of smart cockpit technology, in-vehicle systems have gradually acquired basic capabilities such as environmental adjustment, infotainment, and human-computer interaction. They support functions such as wallpaper changing and ambient lighting adjustment that are manually or triggered by specific scenarios. However, these functions are usually manually set by the user or triggered based on fixed scenarios, making it difficult to dynamically adjust them according to the driver's real-time emotional state, resulting in a poor human-computer interaction experience.

[0003] Currently, some in-vehicle systems can recognize human emotions using visual models, enabling dynamic perception of the cabin environment. However, existing in-vehicle systems rely solely on a single visual sensor for facial detection, resulting in weak capabilities for recognizing complex emotions and consequently, insufficient accuracy in emotion recognition results.

[0004] Therefore, improving the accuracy of emotion recognition in complex in-vehicle environments has become an urgent technical problem to be solved in order to improve the emotional interaction experience of smart cockpits. Summary of the Invention

[0005] In view of the above problems, this disclosure provides a cockpit linkage control method, storage medium, and vehicle to overcome or at least partially solve the above problems. The technical solution is as follows: Acquire facial images of personnel inside the cockpit and determine the facial interference state of the corresponding facial images of the personnel; wherein, the facial interference state is determined by one or more of the following: lighting conditions, degree of facial occlusion, and posture changes; The facial image is input into multiple emotion detection models to obtain the corresponding output detection results; Based on the facial interference state, the evaluation weight corresponding to each emotion detection model is determined, and the detection results are weighted and fused according to the evaluation weight to obtain the emotional state of the person. Based on the emotional state, a cockpit linkage strategy is generated to transition the cockpit from the current state to the target state. Based on the cockpit linkage strategy, at least one environmental control device in the cockpit is controlled to perform progressive linkage actions.

[0006] The evaluation weights for each emotion detection model are determined by analyzing the facial interference state of facial images. Then, the detection results output by multiple emotion detection models are dynamically fused based on the evaluation weights. This allows the model's contribution to the emotion detection results to be adjusted according to its reliability under different facial interference states. As a result, the emotion recognition vector output by weighted fusion is closer to the actual emotions of people, effectively improving the accuracy of emotion recognition in complex in-vehicle environments and avoiding misidentification of emotions by a single model under visual interference.

[0007] In one embodiment, a cockpit linkage strategy for transitioning the cockpit from its current state to a target state is generated, specifically including: Based on a preset sliding time window, the temporal change characteristics corresponding to the emotional state are determined; Based on the temporal change characteristics, an emotional intervention state is determined to transition the cabin from the current state to the target state. Obtain the target cabin environment parameters associated with the emotional intervention state, and generate a cabin linkage strategy based on the target cabin environment parameters.

[0008] By continuously identifying the temporal trends of emotions, we can capture the dynamic evolution of human emotions, thereby more accurately judging the persistence and volatility of emotions. Then, by using the temporal change characteristics, we can determine the emotional intervention state required for the current emotional state, driving the cabin environment to smoothly transition from the current state to a more suitable target state. The resulting cabin linkage strategy is no longer an instantaneous response to the recognition result of a single frame, but a gradual adjustment based on the emotional evolution process. This avoids frequent cabin switching caused by short-term emotional fluctuations and makes environmental changes more in line with the continuous development of user emotions, significantly improving the naturalness and comfort of the interaction.

[0009] In one embodiment, a cockpit linkage strategy is generated based on the target cockpit environment parameters, specifically including: Obtain the current parameter values ​​of each environmental control device in the cockpit; The control parameter sequence corresponding to the transition from the current parameter value to the target cabin environment parameter is calculated using a preset interpolation algorithm. Based on the sequence of control parameters, a corresponding cockpit linkage strategy is generated.

[0010] By generating a continuous sequence of control parameters through interpolation algorithms, the cabin environment can achieve a gradual transition from the current parameter value to the target cabin environment parameter under emotional drive. This effectively avoids the perceptual conflict caused by the sudden change of environmental parameters in traditional real-time control, making the changes in cabin atmosphere more synchronized with the evolution of user emotions. At the same time, the continuous and controllable adjustment process also reduces the possible attention interference to the driver, significantly improving the smoothness of interaction and user comfort.

[0011] In one embodiment, a preset interpolation algorithm is used to calculate the control parameter sequence corresponding to the transition from the current parameter value to the target cabin environment parameter, specifically including: Based on the vehicle's driving status information, determine the driving risk index corresponding to the current vehicle; Based on the driving risk index, the transition time corresponding to the transition of the cabin from the current parameter value to the target cabin environment parameter is determined; wherein, the transition time and the driving risk index are negatively correlated. By using a preset interpolation algorithm, the control parameters corresponding to each time point in the transition duration are determined, so as to obtain the control parameter sequence when smoothly transitioning from the current parameter value to the target cabin environment parameter based on the control parameters.

[0012] By calculating the control parameters at each time point within the transition period, the smoothness of the cabin environment change process is ensured, reducing the abruptness of cabin environment switching and improving the overall interactive experience. Simultaneously, by correlating the driving risk index with the transition period of cabin environment adjustment, the system can adaptively adjust the urgency of cabin linkage behaviors under different driving loads. By dynamically adjusting the transition period and the adjustment range of cabin parameters, a balance between driving safety and user experience can be achieved in the linkage actions of the cabin under different driving conditions.

[0013] In one embodiment, before generating the corresponding cockpit linkage strategy based on the control parameter sequence, the method further includes: Determine whether the driving risk index meets the preset linkage intervention conditions; If not, the control parameter sequence is adjusted so that the intensity of the action corresponding to the adjusted control parameter sequence is adapted to the driving risk index.

[0014] When the vehicle's driving risk index does not meet the conditions for joint intervention, the intensity of the action of the environmental adjustment device in the cabin is adaptively adjusted. This allows the system to actively suppress the magnitude of changes in the cabin environment when the driving risk is high, ensuring that all cabin linkage actions are kept within a low interference intensity range. This avoids the attention distraction problem that may be caused by a fixed intensity of cabin response in high-risk scenarios, thereby ensuring driving safety.

[0015] In one embodiment, the detection result refers to an emotion probability vector composed of confidence scores corresponding to multiple emotion dimensions. The detection result is weighted and fused according to the evaluation weights to obtain the person's emotional state, specifically including: Based on the evaluation weights, the emotion probability vectors are weighted and fused to obtain the corresponding emotion recognition vectors; Multimodal environmental parameters reflecting the emotional state of the person are collected, and the emotion recognition vector is corrected based on the multimodal environmental parameters to obtain the corrected emotion recognition vector. The emotion type corresponding to the emotion dimension with the highest confidence in the modified emotion recognition vector is taken as the emotional state of the person.

[0016] By fusing multimodal environmental parameters to correct the initially identified emotion recognition vector, the accuracy of emotion recognition can be further improved, effectively compensating for the errors that may exist in single facial image detection. The corrected emotion recognition vector not only reflects the current emotional state of the person, but also enhances the reliability of the emotion detection results through the redistribution of confidence, thereby ensuring that the subsequent cabin linkage strategy is more in line with actual needs.

[0017] In one embodiment, after obtaining the emotional state of the person, the method further includes: The dominant emotion of the group is determined based on the emotional state and decision weight of each individual; wherein the decision weight is determined based on at least one of the individual's cabin location, age, and gender. By responding to the dominant emotions of the group, a cockpit linkage strategy corresponding to the cockpit is generated.

[0018] By assigning differentiated decision weights to individuals based on their emotional states and roles within the cabin, the system can determine the dominant direction of group emotions in multi-person scenarios. This allows for the generation of cabin interaction strategies that better meet the needs of collaborative action, avoiding the limitation of the cabin environment only responding to the emotions of a single individual while ignoring the feelings of other members. Under the premise of ensuring driving safety, this enhances the overall interactivity of cabin emotional interaction.

[0019] In one embodiment, a cockpit-related linkage strategy is generated in response to the dominant group emotion, specifically including: Based on the emotional state of the driver among the personnel, determine the corresponding emotional response type of the cockpit; Based on the emotional response type, a cockpit linkage strategy corresponding to the cockpit is generated in response to the driver's emotional state or the dominant group emotion.

[0020] By determining the type of emotional response in the cockpit, we can decide whether the cockpit linkage strategy should prioritize responding to the driver's emotions or comprehensively consider the dominant emotions of the group. This allows us to take into account the emotional needs of other passengers while ensuring driving safety, making the generation of cockpit linkage strategies more adaptable to different scenarios and more human-centered.

[0021] A computer-readable storage medium storing computer-executable instructions for use in a vehicle control system, wherein the computer-executable instructions are configured as follows: The cockpit linkage control method described in any of the above items.

[0022] A vehicle, characterized in that it comprises: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enable the at least one processor to: Perform the cockpit linkage control method as described in any of the preceding items.

[0023] By employing the above technical solutions, this disclosure provides a seat position adjustment method, storage medium, and vehicle. It determines the evaluation weight of each emotion detection model based on the facial interference state of the facial image, and then dynamically fuses the detection results output by multiple emotion detection models according to the evaluation weight. This allows for adjustment of the model's contribution to the emotion detection results based on its reliability under different facial interference states, making the final weighted fusion output emotion recognition vector closer to the actual emotions of the person. This effectively improves the accuracy of emotion recognition in complex in-vehicle environments and avoids misidentification of emotions by a single model under visual interference.

[0024] The above description is merely an overview of the technical solution disclosed herein. In order to better understand the technical means of this disclosure and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this disclosure more apparent and understandable, specific embodiments of this disclosure are described below. Attached Figure Description

[0025] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this disclosure. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A flowchart illustrating a cockpit linkage control method provided in an embodiment of this application is shown; Figure 2The figure shows a schematic structural diagram of a cockpit linkage control device provided by an embodiment of the present application; Figure 3 The figure shows a schematic structural diagram of a vehicle provided by an embodiment of the present application. Detailed implementation manners

[0026] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be fully communicated to those skilled in the art.

[0027] Based on the above application scenarios, to solve the technical problem of how to improve the accuracy of emotion recognition in a complex in-vehicle environment, the present application provides a cockpit linkage control method, as Figure 1 shown Figure 1 is a schematic flowchart of a cockpit linkage control method provided by an embodiment of the present application. This method can be applied to an in-vehicle control system. The method includes: S101: Collect the facial images of the people in the cockpit and determine the facial interference state of the facial images corresponding to the people; wherein, the facial interference state is determined by one or more of the illumination condition, the degree of facial occlusion, and the posture change.

[0028] Cockpit linkage control means that the in-vehicle control system, based on the real-time perception of the people's states, coordinately adjusts multiple environmental adjustment devices in the cockpit, such as ambient lights, aromatherapy, audio, display screen themes, etc., to provide an immersive sensory experience for the people. In a complex and changeable in-vehicle environment, the traditional manual setting or fixed-scene triggering method for cockpit perception adjustment is difficult to meet the dynamic needs of people for emotional interaction. Therefore, the embodiment of the present application collects the facial images of the people in the cockpit, detects the emotions of the facial images, clarifies the emotion state corresponding to the current people, and then controls the cockpit environment to perform adaptive adjustment according to the emotion state of the people. Compared with the traditional manual adjustment method, it can realize the automatic perception of the emotion state and the intelligent linkage of the cockpit environment, and further improve the emotional interaction ability of the cockpit.

[0029] "Personnel" refers to the driver or passengers inside the vehicle. To ensure driving safety, when adjusting the cabin environment, the driver's emotional state must be considered first to ensure that emotional fluctuations do not negatively impact driving behavior. Therefore, when multiple personnel are detected in the cabin, the driver's facial image is captured by default. If the driver is not detected in the driver's seat, the facial images captured are those of passengers in other positions within the cabin. These facial images are captured non-contactly through an Occupant Monitoring System (OMS). As a comprehensive monitoring platform in the intelligent vehicle cabin used for real-time perception, identification, and analysis of the status of all personnel inside the vehicle, OMS can capture facial images of occupants through cameras or other image acquisition devices. To ensure the accuracy of emotion recognition, the system first preprocesses the facial images. First, multi-scale Retinex illumination correction is performed on the original facial images. By equalizing the brightness of the facial images, the visibility of key facial regions (eyes, mouth, eyebrows) under low light or overexposure conditions is enhanced. Then, the stability of the facial image quality is detected. If the key points in multiple consecutive frames of facial images are unstable, it indicates that the currently acquired facial image may have local facial occlusion or pose changes. Once this situation is identified, a first-order Kalman filter will be used to smooth the emotion confidence sequence detected by the model when performing emotion detection on the facial images in the future, thereby suppressing label jitter.

[0030] The quality of captured facial images can degrade due to environmental or personal factors, directly impacting the accuracy of emotion recognition algorithms. For instance, vehicles frequently experience drastic lighting changes during actual driving, such as entering and exiting tunnels, meeting oncoming traffic at night, and being shaded by trees. Ambient lighting can cause facial features to be overexposed, lost, or hidden in shadows. People may wear sunglasses, masks, or adopt postures such as turning their heads to the side or looking down, resulting in missing key facial information or distorted facial geometric features. Traditional single-vision model detection methods, which rely solely on simple visual analysis of facial images, often fail to guarantee accuracy in emotion recognition under the influence of real-world driving interference. Therefore, this application's embodiments introduce multiple emotion detection models and combine them with an evaluation of facial interference states in the facial images, dynamically fusing the detection results output by each model to effectively improve the reliability of emotion recognition in complex in-vehicle environments.

[0031] Facial interference states refer to a set of interfering factors that affect facial image quality and the accuracy of emotion recognition. Specifically, they are determined by one or more of the following: lighting conditions, facial occlusion level, and pose changes. Regarding lighting conditions, the system calculates a lighting interference index based on the brightness distribution and contrast of the facial image. If overexposed areas or dark areas exceed a preset threshold, the image is classified as having high lighting interference. For facial occlusion, the system assesses the occlusion level by analyzing the visibility ratio of facial key points. For example, if more than 30% of facial key points are not detected, the system will mark it as a high occlusion state. Pose changes are quantified by calculating the deviation between the facial angle and the standard frontal face angle. If the yaw angle, pitch angle, or roll angle exceeds a certain range, it is considered a high pose change state.

[0032] S102: Input the facial image into multiple emotion detection models and obtain the corresponding output detection results.

[0033] In this embodiment, a multi-level emotion detection model is used for fusion recognition. After the acquired facial image is input into multiple emotion detection models, each emotion detection model outputs a corresponding detection result. Typically, the reliability of multi-level emotion detection models varies under different facial interference conditions. Taking a three-level emotion detection model as an example, the emotion detection model can be the ML Kit face detection model, Face Mesh, or the TFLite lightweight CNN model. When performing emotion detection on facial images, these three types of models need to be used sequentially to recognize the facial image. Both ML Kit and TFLite are open-source face detection models. ML Kit focuses solely on smile and eye opening / closing recognition, achieving high accuracy under good lighting conditions without strong glare. Face Mesh is a model based on facial landmark tracking, demonstrating greater stability when handling partial facial occlusion or pose changes. TFLite is a lightweight convolutional neural network model capable of efficient emotion recognition in environments with limited computing resources. When the output confidence of both ML Kit and Face Mesh is low, TFLite can serve as a supplementary detection method to further improve the coverage and accuracy of emotion recognition.

[0034] The ML Kit face detection model can perform emotion recognition on facial images by calling a preset face detection interface. Since the ML Kit face detection model only provides detection services for the emotion of "happiness," if the final output parameter `isSmilingProbability` is greater than a certain threshold after calling the face detection interface to recognize the facial image, it indicates that the person's facial emotion state is happy. Face Mesh identifies the emotion state based on the extracted facial key points according to preset geometric rules, as shown in Table 1, which includes some parameter examples of the geometric rules. Table 1

[0035] As shown in Table 1, some geometric rules can be used to determine a person's emotional state based on the geometric relationship of key points. For each emotional state, the matching degree between the key point and the corresponding rule can be calculated based on the actual value of the key point and the threshold defined in the rule. For example, for the rule "eyebrow-eye distance < 0.06", the actual value is 0.05, which is less than 0.06. The confidence level corresponding to this value is calculated as 1 minus the ratio between the actual value and the threshold. When the actual value is much smaller than the threshold, the confidence level is close to 1. For the emotion type of anger, both the eyebrow-eye distance and eye opening / closing degree need to be satisfied. The final matching degree can be the product of the confidence levels of the two sub-rules or the minimum value. Table 1 only provides one possible rule example. The specific rule parameters can be set according to actual needs, and this application does not limit them.

[0036] TFLite is a lightweight Convolutional Neural Network (CNN) model. After processing facial images for grayscale and size, the images are input into the TFLite model, which directly outputs the softmax probabilities of various emotions. In this embodiment, the emotion categories adopt the seven emotions defined by the standard FER2013: anger, disgust, fear, happiness, sadness, surprise, and neutral, with corresponding indices of 0-6. Therefore, TFLite directly outputs the probabilities corresponding to each of the seven emotions. To facilitate the subsequent fusion of detection results, the detection results output by the ML Kit face detection model and FaceMesh are also in the form of seven-dimensional vectors, i.e. Each dimension in the vector corresponds to an emotion type. Since the ML Kit face detection model only provides high-confidence outputs for the emotion of happiness, if it detects an isSmilingProbability greater than a certain threshold, then the dimension corresponding to the emotion of happiness... Then set it to 1, and set all other dimensions to 0, that is... Face Mesh determines which emotion type a key point in the current facial image matches based on preset geometric rules, and further calculates the matching degree between the key point and the emotion type. The matching degree is assigned to the corresponding emotion dimension, while all other emotion dimensions are set to 0. If no judgment rule is triggered, the detection result is set to an all-zero vector or a uniformly distributed vector. .

[0037] S103: Based on the facial interference state, determine the evaluation weight corresponding to each emotion detection model, and perform weighted fusion of the detection results according to the evaluation weight to obtain the emotional state of the person.

[0038] After completing the recognition of the multi-level emotion detection model, it is necessary to assign different evaluation weights to each emotion detection model based on the facial interference state of the facial image. According to the evaluation weights, the detection results output by each emotion detection model are dynamically weighted and fused. By dynamically adjusting the contribution of the output results of each emotion detection model, the overall accuracy of emotion recognition can be effectively improved, and the emotional state of the person can be obtained.

[0039] The system dynamically adjusts the weights of each emotion detection model to address different facial interference conditions. For example, under high-light interference, the weight of the ML Kit face detection model is appropriately reduced, while the weights of the Face Mesh and TFLite lightweight CNN models are increased to compensate for the impact of lighting on recognition accuracy. Similarly, under conditions of high occlusion or high pose variation, Face Mesh is given a higher weight due to its robustness to partial occlusion and pose changes, thereby improving the overall reliability of emotion recognition. When the output confidence of both the ML Kit face detection model and Face Mesh is low, the TFLite lightweight CNN model will serve as the primary detection method, and its evaluation weight will be automatically increased to ensure that it can still output relatively reliable emotion recognition results under complex interference conditions. Through this dynamic weighted fusion mechanism, the system can more flexibly cope with the complex and ever-changing in-vehicle environment, avoid misjudgments caused by external interference from a single model, and fully utilize the complementarity between multiple emotion detection models to improve the overall accuracy and stability of emotion recognition.

[0040] After weighted fusion of the detection results from each model, a comprehensive emotion recognition vector is obtained. This vector represents the current emotional state distribution of the individual. Based on the probability distribution of various emotion types within this vector, the individual's dominant emotional state is determined. Since the emotion recognition vector is a seven-dimensional vector, the emotion type corresponding to the element with the largest value is the final identified emotional state. It should be noted that to ensure the accuracy and stability of the emotion recognition results, further post-processing of the emotion recognition vector is required. This involves performing time-series analysis on the emotion recognition vectors from multiple consecutive frames, combined with a first-order difference algorithm to calculate the amplitude of emotional state changes between adjacent frames. If the amplitude of changes is too large and lacks reasonable contextual support, it is considered an abnormal fluctuation. In this case, a first-order Kalman filter is used to smooth the emotion confidence sequence, thereby filtering out abnormal fluctuations that may be caused by transient interference.

[0041] S104: Based on the emotional state, generate a cockpit linkage strategy to transition the cockpit from the current state to the target state.

[0042] After identifying the emotional state of the passengers, the cabin environment can be adaptively adjusted based on this emotional state. This enables the coordinated operation of multiple environmental adjustment devices within the cabin, providing passengers with a more comfortable and immersive experience. The target state refers to the ideal cabin environment intended to adapt to the current emotional trend of the passengers. Essentially, it is the set of cabin parameters expected to be achieved after the cabin has been adjusted by the environmental adjustment devices. For example, if it is determined that anger has persisted for a period of time, it is necessary to consider adjusting the cabin environmental parameters to soothe the passengers' emotions. The cabin linkage strategy adopted could be to adjust the ambient lighting to a cool blue tone, play soothing white noise, and switch the user interface to a low visual load theme. By controlling the corresponding environmental adjustment devices within the cabin to execute the control commands corresponding to the above cabin linkage strategy, the cabin can transition from the current state to the more soothing target state. The target state is derived from the temporal characteristics of emotional changes, such as the duration of emotion, the slope of emotional change, and signs of recovery, driven by a finite state machine decision. Emotional persistence refers to the duration of the current emotional state. For example, at a sampling rate of 10Hz, if anger appears continuously for 5 frames, then the emotional persistence metric EP = 0.5s. Emotional change slope refers to the rate of change of the confidence level of an emotional state per unit time. Taking anger as an example, its corresponding emotional change slope can be calculated using the following formula:

[0043] Where N is the sliding time window, The frame interval for facial images, in seconds. and These represent the confidence levels for anger at time T and time TN, respectively.

[0044] The emotional recovery index is used to determine whether negative emotions are subsiding, and it can be calculated using the following formula: ,in, The confidence level of neutral sentiment at time T is represented. This represents the average confidence level of all negative emotions within the most recent time period from time point T-5 to T (e.g., the most recent 0.5 seconds; if each frame is 100ms, then it corresponds to the most recent 5 frames of data). It is calculated by subtracting the average intensity of recent negative emotions from the intensity of current neutral emotions. A positive difference indicates that the current neutral emotion is significantly higher than the recent negative emotion level, and the emotion may be easing. A negative or zero difference indicates that the current neutral emotion does not exceed the recent negative emotion level, and no clear signs of recovery have been detected.

[0045] It should be noted that, in order to ensure the real-time performance and accuracy of cockpit-linked decision-making, the above indicators need to be updated at preset time intervals. Furthermore, when generating cockpit-linked strategies, transient fluctuations in emotional state need to be filtered out. Therefore, based on the emotional recovery index and emotional persistence in the aforementioned time-series change characteristics, it is determined whether the emotional state is a transient disturbance. Assuming RI > 0.2 and EP < 2 seconds, it is determined to be a transient fluctuation and will not trigger adjustments to the cockpit-linked strategy.

[0046] The essence of generating a cockpit linkage strategy is to generate a sequence of control commands that smoothly transition from the current cockpit environment to the aforementioned target state. This strategy uses interpolation algorithms to ensure that all environmental adjustment devices, such as lighting, sound, and visual interfaces, can work together to gradually complete the state transition, thereby avoiding the abruptness and interference caused by sudden changes in emotions or the cockpit environment, making the entire interaction process seem like a natural flow that resonates with the emotions of the person.

[0047] To achieve a more personalized cabin interaction experience, the system also supports emotion preference learning based on historical data. By recording and analyzing users' emotional states and corresponding cabin environment settings over long-term use, the system can gradually build a personalized emotion preference model for each user. This model not only includes users' environmental preferences under different emotional states but also dynamically adjusts recommendation strategies based on their usage habits. For example, some users may prefer bright ambient lighting and upbeat music when feeling sad, while others may prefer soft lighting and a quiet environment. Based on this differentiated preference data, the system will tailor a unique cabin interaction control scheme for each user. Simultaneously, the system supports manual adjustment and feedback mechanisms. Users can easily rate or modify the current cabin environment settings through simple operations. This feedback data will be incorporated into the optimization process of the emotion preference model, further improving the system's intelligence and user experience.

[0048] S105: Based on the cockpit linkage strategy, control at least one environmental control device in the cockpit to perform progressive linkage actions.

[0049] Environmental control devices refer to equipment within the cabin used to alter the physical or sensory attributes of the passenger environment, such as ambient lighting, audio systems, air conditioning, fragrance generators, power seats, and the UI theme of displays or instrument panels. Based on the aforementioned cabin linkage strategy generated according to the occupant's emotional state, the vehicle control system sends corresponding control commands to each environmental control device within the cabin to drive these devices to perform progressive linkage actions, achieving a smooth transition of the cabin environment from the current state to the target state. Progressive linkage actions refer to the coordinated action of each environmental control device gradually adjusting its own parameters according to a preset time sequence, rather than an instantaneous jump to the target cabin environment parameters.

[0050] That is to say, the vehicle control system will generate an accurate sequence of control parameters for each environmental adjustment device according to the cockpit linkage strategy, and form a continuous sequence of control parameters by interpolating the control parameters at multiple time points, ensuring that the actions of the environmental adjustment devices are smooth without jumps. For example, if the target state is to reduce the temperature from the current 26°C to 22°C and the transition time is set to 3 minutes, the control system will calculate the target temperature value per minute or even per second according to the preset temperature change curve, and send continuous adjustment instructions to the air conditioning system to make the temperature drop at a uniform and slow rate, avoiding the discomfort caused by sudden temperature changes. For the adjustment of the ambient light, the same progressive principle is followed. If it is necessary to switch from warm yellow to cold blue, the system will control the light to gradually adjust the values of the RGB color channels during the transition time to achieve a smooth gradient of the hue, and the brightness can also be softly transitioned synchronously as needed. The switching of the user interface also adopts a progressive animation effect, such as the smooth movement of interface elements, the gradual change of transparency or the scaling transition, avoiding the impact on the user's vision caused by abrupt interface jumps. During the entire linkage process, the environmental adjustment devices do not work independently, but maintain the synchronization of actions and the consistency of rhythm through the unified coordination of the vehicle control system. For example, the rate of change of the hue of the ambient light, the adjustment gradient of the air conditioning temperature, and the change amplitude of the audio volume will match each other to jointly create a coordinated and unified environmental atmosphere transition effect. In addition, the system will also monitor the execution process of each device in real time, obtain the actual operating state of the device through the feedback mechanism. If a certain device has a response delay or parameter deviation, the control system will timely adjust the control instructions of other related devices dynamically to ensure the smoothness and accuracy of the overall cockpit environment transition. Through this progressive linkage action based on precise timing control and multi-device collaboration, the adjustment process of the cockpit environment can be well matched with the user's emotional perception, effectively avoiding the sensory abruptness brought by the traditional instantaneous adjustment method, and thus providing a more natural, comfortable and immersive cockpit experience for the user. <�

[0051] In one embodiment, the detection result refers to an emotion probability vector composed of confidence levels corresponding to multiple emotion dimensions. Each emotion probability vector is a seven-dimensional vector, and each vector value represents the output confidence level corresponding to each emotion dimension. According to the order of the indices in the vector, that is, the vector values indicated by indices 0-6, the emotion dimensions are successively represented as seven types of emotions: anger, disgust, fear, happiness, sadness, surprise, and neutral. In the embodiments of the present application, the multi-level emotion detection model adopted is a three-layer progressive architecture. In actual use, the number and type of emotion detection models used can be set according to actual needs, and the present application does not limit this.

[0052] After obtaining the emotion probability vector output by each emotion detection model, it is necessary to identify the facial interference state corresponding to the current facial image. Then, based on a predefined mapping relationship, the evaluation weight corresponding to each emotion detection model under the current facial interference state is determined. Next, the emotion probability vectors output by each emotion detection model are weighted and fused according to the evaluation weights to obtain an emotion recognition vector that incorporates the advantages of each model. For example, when the image illumination intensity is detected to be below a preset threshold, it is determined to be a low-light interference state. In this case, the evaluation weight w1 of the TFLite lightweight CNN model will be set to a higher value, such as 0.6, while the weights w2 and w3 of the ML Kit face detection model and Face Mesh will be adjusted to 0.2 and 0.2 respectively to highlight their recognition advantages under low light conditions. In the ideal state where there is no significant interference or the confidence of each model is high, average weighting or dynamic weight allocation based on the overall performance of the model in historical data can be used, such as w1=0.3, w2=0.4, w3=0.3, to ensure a balanced performance of the advantages of each model. By dynamically adjusting weights based on facial interference, the final comprehensive emotion recognition vector more accurately reflects the person's true emotional state, providing a reliable basis for subsequent cabin interaction strategy generation. The weighted fusion emotion recognition vector is also a seven-dimensional vector. Based on the confidence level corresponding to each emotion dimension in this vector, the person's current emotional state can be determined, thus achieving robust and accurate recognition in complex environments.

[0053] It should be noted that emotion recognition based solely on facial images can still be affected by fleeting expressions, deliberate concealment, or extreme environmental interference, leading to risks of misjudgment or insufficient confidence. To improve the robustness of emotion detection results in real-world driving scenarios, the preliminary emotion recognition vectors identified above need to be further processed by collecting and fusing real-time data from other sensory dimensions. This allows for cross-validation and confidence calibration of the preliminary recognition results, thereby enhancing the accuracy and contextual plausibility of the final judgment.

[0054] Specifically, multimodal environmental parameters reflecting the emotional state of individuals are collected. These parameters need to be strongly correlated with emotions and include voice emotion features, vehicle dynamic features, and environmental context. Voice emotion features can be obtained by analyzing the fundamental frequency, average energy, and specific keywords of the voice in real time through the vehicle microphone to determine tone. Vehicle dynamic features are obtained through the CAN bus by acquiring signals of intense driving behaviors such as longitudinal and lateral acceleration and sudden braking, which can serve as external manifestations of emotions. Environmental context includes light intensity, timestamps, and road congestion index. This information provides contextual explanations for emotions; for example, congested road sections during evening rush hour are more likely to trigger feelings of irritability.

[0055] After acquiring the multimodal environment parameters, a multimodal input vector can be constructed based on these parameters and the emotion recognition vector obtained from facial image recognition. ; in, Represents the emotion recognition vector. Represents the speech emotion feature vector. Indicates the dynamic characteristics of a vehicle. Indicates the environment context.

[0056] The constructed multi-model input vector is fed into a lightweight gated fusion network (GFU). The GFU then corrects the confidence scores for each emotion dimension in the emotion recognition vector, resulting in a corrected emotion recognition vector. The specific processing logic is as follows:

[0057] in, , The facial occlusion rate can be calculated based on the missing rate of key points in the Face Mesh. For audio feature mapping function, For driving behavior correction function, To adjust the weights, we can take... , For the preset correction vector, This represents the emotion recognition vector. All the parameters mentioned above were determined through offline calibration and do not require online training.

[0058] The corrected emotion recognition vector can be obtained using the gated fusion network. The emotion type corresponding to the emotion dimension with the highest confidence in this vector is the most likely emotional state of the person at present.

[0059] In one embodiment, traditional cockpit response methods are instantaneous mappings from emotional states to adjustment actions, lacking perception of the evolving trends of human emotions and gradual intervention. This results in abrupt adjustments to the cockpit environment, impacting the user's sensory experience. Furthermore, triggering responses based solely on single recognition results can easily lead to frequent cockpit switching due to brief emotional fluctuations, interfering with user attention. Therefore, to further optimize the effectiveness of cockpit linkage control, this application introduces an emotion temporal analysis mechanism to capture the dynamic trends of human emotional states. By analyzing the temporal characteristics of emotional state changes, a smooth transition cockpit linkage strategy is generated to facilitate the transition from the current state to the target state, ensuring a more natural and fluid adjustment process for the cockpit environment.

[0060] After identifying an individual's emotional state, it's necessary to analyze its dynamic evolution characteristics by considering its temporal trends to more accurately grasp the development of their emotions. During time-series analysis, the system employs a sliding time window to continuously track changes in emotional state. For example, setting the sliding time window to 10 seconds, with each slide lasting 2 seconds, extracts features from the emotional state and confidence level within the window, ultimately identifying three temporal change features: emotional persistence, emotional change slope, and emotional recovery index. The size of the sliding time window can be dynamically adjusted based on the specific application scenario to balance the requirements of real-time performance and stability. For instance, in highly dynamic driving scenarios, the sliding window size can be appropriately reduced to improve system sensitivity and respond quickly to changes in emotions; while in relatively stable driving environments, increasing the sliding window size helps to further suppress noise interference and enhance the reliability of emotion recognition results.

[0061] After acquiring the temporal change characteristics, and combining them with preset state transition rules, a finite state machine is driven to determine the most suitable emotional intervention state. The emotional intervention state is used to drive the cabin from the current state to the target state. The emotional intervention state is determined by analyzing the temporal change trend of the emotional state to determine the appropriate form of adjustment for the current cabin environment, including soothing, deep intervention, and maintaining. Assuming EP ≥ 1.0 seconds and ES_angry > 0.1 / second, the emotional intervention state is determined to be soothing (s_calm); if EP ≥ 3.0 seconds and RI < 0.1, the emotional intervention state is deep intervention (s_deep_calm); if RI > 0.25 and EP < 1.5 seconds, the corresponding emotional intervention state is maintaining the current state or reverting to a neutral state (s_neutral).

[0062] A finite state machine defines the cockpit environment as a finite set of states, i.e. Each A set of cabin environment parameters corresponding to different emotional intervention states, for example, These represent the wallpaper identifier, theme color, ambient light hue, and music type, respectively. The cabin environment parameters can be customized and extended to include parameters such as fragrance concentration and ambient light brightness. Therefore, after determining the desired emotional intervention state for the current cabin, the target cabin environment parameters associated with that state can be obtained from the finite state machine. Based on the target cabin environment parameters, a specific cabin linkage strategy can be generated. The cabin linkage strategy is the core scheme guiding the coordinated operation of various environmental control devices, including the control parameters required for each device, as well as the timing logic and transition duration of the actions.

[0063] In one embodiment, the process of generating a cabin linkage strategy involves converting the target cabin environmental parameters into a set of control parameters that can be sequentially executed by various environmental control devices within the cabin. First, the current parameter values ​​of each environmental control device in the cabin are obtained via the vehicle bus or the control interfaces of each device. Examples include the current RGBW brightness values ​​of each channel of the ambient lighting, the current output volume and source identifier of the audio system, and the current set temperature and fan speed of the air conditioning. Then, the target cabin environmental parameters of each environmental control device are compared with the obtained current parameter values. For each set of environmental parameters, a preset interpolation algorithm is used to calculate the corresponding control parameter sequence when transitioning from the current parameter value to the target cabin environmental parameters. The interpolation algorithm used here can be the Bessel interpolation algorithm, which, combined with the transition time dynamically determined by the driving risk index, generates a discrete time value sequence, i.e., the control parameter sequence. The control parameter sequence clearly defines the intermediate value that the environmental control device needs to achieve at each time point from the start to the end within the total transition time. In this way, a smooth, abrupt parameter change trajectory is generated for each environmental control device. Finally, based on the control parameter sequence, a corresponding cockpit linkage strategy is generated. This strategy indicates the specific target values ​​for the coordinated actions of each environmental control device at each moment within the total transition time. Once generated, the cockpit linkage strategy is sent to the cockpit domain controller or the local controllers of each device, driving them to execute sequentially according to this control parameter sequence. This ensures that the entire cockpit environment can evolve smoothly, gradually, and seamlessly from the current state to the target state, achieving a smooth transition in the emotional interaction and effectively avoiding interference that may be caused by parameter jumps.

[0064] In one embodiment, during cockpit linkage control, directly causing each environmental adjustment device to jump from its current parameters to the target cockpit environmental parameters would result in abrupt and jarring changes in the cockpit environment. Such instantaneous changes not only disrupt the continuity of the driver's emotions but may also distract the driver during driving. Therefore, this application embodiment introduces a dynamic interpolation control mechanism based on driving risk to transform the abstract cockpit linkage strategy into a smooth sequence of actions that can be safely executed.

[0065] Specifically, the driving risk index corresponding to the current vehicle is determined based on the vehicle's driving status information. Driving status information includes vehicle speed, road type, and congestion index. According to predefined mapping rules, this driving status information can be mapped to corresponding risk component values. The vehicle speed risk component can be calculated using a piecewise function; for example, the risk component is 0.2 when the vehicle speed is below 30 km / h, linearly increasing to 0.5 between 30-80 km / h, and directly set to 0.9 when the speed is above 80 km / h. In the road type risk component, the risk of highways is significantly greater than that of urban expressways and residential roads. The congestion index is obtained through a real-time traffic data interface, with a congestion level of 0-100% mapped to a risk component value of 0.1-0.9. The three risk component values ​​are then weighted and summed according to preset weights to obtain the comprehensive driving risk index (DRI), whose value range is normalized to the [0,1] interval.

[0066] Then, based on this driving risk index, the total duration of the cabin environment parameter transition is dynamically adjusted. Specifically, it can be calculated using the following formula: The transition time is negatively correlated with the driving risk index. When the driving risk index is high, such as when the vehicle is traveling at high speed or in complex traffic flow, the system will automatically shorten the transition time of environmental changes to quickly complete the cabin environment adjustment and reduce the need for driver attention. When the driving risk index is low, a longer transition time is allowed.

[0067] Finally, the control parameters corresponding to each time point in the transition period are determined using a preset interpolation algorithm. The Bessel interpolation algorithm takes the initial value, target value, and determined transition period of the parameter as input, and calculates the intermediate value that the parameter should reach at each tiny time interval throughout the entire transition period, thereby generating a parameter trajectory that changes continuously and smoothly over time, i.e., the control parameter sequence. This process can be represented as:

[0068]

[0069] in, Indicates the current parameter value. This represents the target cockpit environmental parameters, and t represents the time elapsed since the start of the transition. Indicates the transition duration. It is a cubic Bessel easing function.

[0070] After obtaining the control parameters corresponding to each time point using the above interpolation algorithm, a continuous sequence of control parameters can be formed throughout the entire transition time. This sequence of control parameters ensures that the state transitions of all devices within the cockpit are no longer instantaneous, but rather gradually adjusted according to the control parameters. This makes the changes in the cockpit environment conform to the rhythm of human perception, avoiding visual, auditory, or tactile shocks caused by sudden parameter changes.

[0071] In one embodiment, all linkage actions of the environmental control device only take effect when the driving risk index is low. When the driving risk index is high, the driver's attention needs to be highly focused on core driving tasks such as road condition judgment, vehicle control, and emergency response. Any dynamic changes in the cabin environment may distract the driver's limited attention resources and increase driving safety hazards. Therefore, before generating the cabin linkage strategy, it is necessary to determine whether the driving risk index meets the preset linkage intervention conditions. The linkage intervention condition is a predefined safety threshold, such as DRI ≤ 0.7. This condition marks the safety boundary that allows the system to execute the full cabin linkage function. The real-time calculated DRI is compared with this linkage intervention condition. If the DRI meets this condition, it indicates that the current driving environment is relatively safe, and it will be determined that the cabin linkage strategy can be generated according to the original control parameter sequence. Conversely, if the driving risk index does not meet the preset intervention conditions, it indicates that the vehicle is in a high-risk driving scenario. In this case, the control parameter sequence will be adjusted. This adjustment is not a simple prohibition of linkage, but rather follows the principle of matching the action intensity with the driving risk index. The pre-calculated control parameter sequence is suppressed so that the action intensity corresponding to the adjusted control parameter sequence matches the driving risk index. Action intensity refers to the change range of cabin environment parameters per unit time, such as the rate of change of ambient light color temperature difference, the adjustment step size of music volume, and the release intensity of fragrance concentration. Specific adjustment methods include, but are not limited to: prohibiting wallpaper switching and only allowing wallpaper preloading; limiting the brightness of ambient light to no more than 20% and disabling dynamic mode; limiting music volume to no more than 20% and prohibiting song switching and voice announcements; freezing changes to the UI theme color.

[0072] Through the above adjustments, the overall action intensity of the adjusted control parameter sequence is now matched with the current high driving risk index, ensuring that all changes in the cabin environment are constrained to a low-interference, low-intensity range in high-risk scenarios. This eliminates the potential risk of distracting the driver's attention due to drastic or frequent changes in the cabin environment, thus ensuring driving safety.

[0073] In one embodiment, the implementation described above is geared towards the driver. However, in real-world smart cockpit applications, multiple people are often present in the vehicle. The identification and response to a single driver's emotions may not accurately reflect the overall riding atmosphere, and could even lead to a conflict between the cockpit environment adjustment and the feelings of some passengers. To address the issue of the coordination of emotional interaction in multi-person scenarios, this application embodiment, after completing individual emotion identification, performs collaborative perception and decision-making on group emotions, thereby improving the overall inclusiveness of cockpit emotion adjustment while ensuring driving safety.

[0074] Specifically, the dominant group emotion is determined based on the individual emotional state and decision-making weight of each person in the cabin. The dominant group emotion refers to the comprehensive emotional state that represents the emotional tendencies of the majority or key personnel in the cabin, and it is the core basis for adjusting the group-based cabin environment. The system simultaneously collects facial images of the driver and other passengers using OMS wide-angle cameras or distributed cameras, and independently outputs the emotion label and confidence level of each person using the same three-level emotion recognition process. Subsequently, differentiated decision-making weights are preset based on at least one factor among the person's cabin location, age, and gender. For example, because the driver's emotions directly affect driving safety, they are given the highest weight, followed by the front passenger, and then relatively lower weights for rear passengers. Alternatively, the decision-making weight of elderly passengers can be appropriately increased to prioritize their comfort.

[0075] Based on the assigned decision weights, the emotion recognition results of all individuals are weighted and fused to calculate the dominant group emotion that represents the overall emotional tendency within the vehicle. This process is represented as follows:

[0076] in, Indicates decision weights, Indicates emotional state. As an indicator function, it is in The value is 1 if the condition is met, and 0 otherwise. represents the confidence level, and k represents the current sentiment category involved in the calculation.

[0077] As shown in the formula above, when calculating the dominant emotion of the group, the above calculation process needs to be performed for each emotion category. k represents the emotion category involved in the calculation in the current round. Based on whether each person identifies the existence of that emotion category, the specific value of the indicator function is determined. Then, the indicator function value and confidence level are weighted according to the decision weight. After performing the above weighting process for all people and summing the results, the total score corresponding to each emotion category can be calculated. Ultimately, the emotion category with the highest total score is the dominant emotion of the group. After determining the dominant emotion in the cabin, the vehicle control system can generate cabin linkage strategies based on the dominant emotion to achieve precise adaptation of the cabin environment to the overall emotional atmosphere inside the vehicle.

[0078] In one embodiment, the linkage control of the intelligent cockpit needs to respond to the driver's emotions to ensure driving safety, while also considering the feelings of other passengers to improve the overall experience. Therefore, when generating the final cockpit linkage strategy, the vehicle control system needs to determine whether to generate the strategy in response to the emotional state of a single person or the dominant emotions of the group. This allows for the selection of the most suitable linkage strategy for the current situation in complex multi-person in-vehicle environments.

[0079] First, based on the driver's emotional state, the corresponding emotional response type for the cabin needs to be determined. The emotional response type refers to the decision category used to determine the basis for generating cabin linkage strategies, including safety-priority type and atmosphere-coordination type. The core is to distinguish whether, in the current scenario, cabin environment adjustment should prioritize ensuring driving safety or focus on coordinating the emotional atmosphere of all occupants. When the driver's emotional state is identified as strongly related to driving safety, such as anger, sadness, or fear, the emotional response type is safety-priority. If the driver's emotional state is identified as a normal emotional state that does not directly threaten driving safety, such as happiness, disgust, or neutrality, the emotional response type is atmosphere-coordination. Based on the emotional response type, the corresponding cabin linkage strategy can be generated by responding to the driver's emotional state or the dominant group emotion. Specifically, if the emotional response type is a safety-priority type strongly related to driving safety, it will prioritize and only respond to the driver's individual emotion to ensure that any safety intervention measures can accurately and quickly act on the driver, ensuring driving safety. For emotional responses that primarily serve to create a positive atmosphere, such as happiness or surprise, the cabin interaction strategy is generated based on the dominant emotion of the group. When there is a conflict between the driver's and passengers' emotions, such as the driver being angry while the passengers are happy, the priority is to appease the driver from a safety perspective, but a neutral approach is preferred in terms of atmosphere selection to balance the experience of different people.

[0080] It should be noted that, considering user privacy, when waving goodbye to other people, users can choose whether to configure multi-person emotion linkage in the cockpit settings interface of the application or the in-vehicle terminal. When this configuration is selected, privacy authorization is enabled by default, and the function of collecting and analyzing the emotion data of all people in the vehicle is enabled.

[0081] In addition, this application also provides a computer-readable storage medium storing computer-executable instructions for use in a vehicle control system, wherein the computer-executable instructions are configured as the cockpit linkage control method provided in this application.

[0082] like Figure 2 As shown, Figure 2 This is a schematic diagram of a cockpit linkage control device provided in an embodiment of this application. The device includes: The acquisition module 201 is used to acquire facial images of people in the cockpit and determine the facial interference state of the facial images of the people; wherein, the facial interference state is determined by one or more of the following: lighting conditions, degree of facial occlusion, and posture changes. The emotion detection module 202 is used to input facial images into multiple emotion detection models and obtain corresponding output detection results; The fusion module 203 is used to determine the evaluation weight corresponding to each emotion detection model based on the facial interference state, and to perform weighted fusion of the detection results according to the evaluation weight to obtain the emotional state of the person. The strategy generation module 204 is used to generate a cockpit linkage strategy based on the emotional state to transition the cockpit from the current state to the target state. The control module 205 is used to control at least one environmental control device in the cockpit to perform progressive linkage actions based on the cockpit linkage strategy.

[0083] Optionally, based on the emotional state, a cockpit linkage strategy is generated to transition the cockpit from the current state to the target state, specifically including: Based on a preset sliding time window, the temporal change characteristics corresponding to the emotional state are determined; Based on the temporal change characteristics, an emotional intervention state is determined to transition the cabin from the current state to the target state. Obtain the target cabin environment parameters associated with the emotional intervention state, and generate a cabin linkage strategy based on the target cabin environment parameters.

[0084] Optionally, based on the target cockpit environment parameters, a cockpit linkage strategy is generated, specifically including: Obtain the current parameter values ​​of each environmental control device in the cockpit; The control parameter sequence corresponding to the transition from the current parameter value to the target cabin environment parameter is calculated using a preset interpolation algorithm. Based on the sequence of control parameters, a corresponding cockpit linkage strategy is generated.

[0085] Optionally, a preset interpolation algorithm is used to calculate the control parameter sequence corresponding to the transition from the current parameter value to the target cabin environment parameter, specifically including: Based on the vehicle's driving status information, determine the driving risk index corresponding to the current vehicle; Based on the driving risk index, the transition time corresponding to the transition of the cabin from the current parameter value to the target cabin environment parameter is determined; wherein, the transition time and the driving risk index are negatively correlated. By using a preset interpolation algorithm, the control parameters corresponding to each time point in the transition duration are determined, so as to obtain the control parameter sequence when smoothly transitioning from the current parameter value to the target cabin environment parameter based on the control parameters.

[0086] Optionally, before generating the corresponding cockpit linkage strategy based on the control parameter sequence, the method further includes: Determine whether the driving risk index meets the preset linkage intervention conditions; If not, the control parameter sequence is adjusted so that the intensity of the action corresponding to the adjusted control parameter sequence is adapted to the driving risk index.

[0087] Optionally, the detection result refers to an emotion probability vector composed of confidence scores corresponding to multiple emotion dimensions. The detection result is weighted and fused according to the evaluation weights to obtain the person's emotional state, specifically including: Based on the evaluation weights, the emotion probability vectors are weighted and fused to obtain the corresponding emotion recognition vectors; Multimodal environmental parameters reflecting the emotional state of the person are collected, and the emotion recognition vector is corrected based on the multimodal environmental parameters to obtain the corrected emotion recognition vector. The emotion type corresponding to the emotion dimension with the highest confidence in the modified emotion recognition vector is taken as the emotional state of the person.

[0088] Optionally, after obtaining the emotional state of the person, the method further includes: The dominant emotion of the group is determined based on the emotional state and decision weight of each individual; wherein the decision weight is determined based on at least one of the individual's cabin location, age, and gender. By responding to the dominant emotions of the group, a cockpit linkage strategy corresponding to the cockpit is generated.

[0089] Optionally, by responding to the dominant emotion of the group, a cockpit linkage strategy corresponding to the cockpit is generated, specifically including: Based on the emotional state of the driver among the personnel, determine the corresponding emotional response type of the cockpit; Based on the emotional response type, a cockpit linkage strategy corresponding to the cockpit is generated in response to the driver's emotional state or the dominant group emotion.

[0090] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0091] Figure 3 This is a schematic diagram of the structure of a vehicle provided in an embodiment of this application.

[0092] For example, such as Figure 3 As shown, the vehicle includes a memory 301 and a processor 302. The memory 301 stores executable program code 3011, and the processor 302 is used to call and execute the executable program code 3011 to perform the cockpit linkage control method.

[0093] This embodiment can divide the vehicle into functional modules according to the above method example. For example, each function can be assigned to a separate module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0094] When each functional module is divided according to its corresponding function, the vehicle may include: The acquisition module is used to acquire facial images of people in the cockpit and determine the facial interference state of the corresponding facial images of people; wherein, the facial interference state is determined by one or more of the following: lighting conditions, degree of facial occlusion, and posture changes. The emotion detection module is used to input facial images into multiple emotion detection models and obtain the corresponding output detection results; The fusion module is used to determine the evaluation weight of each emotion detection model based on the facial interference state, and to perform weighted fusion of the detection results according to the evaluation weight to obtain the emotional state of the person. The strategy generation module is used to generate cockpit linkage strategies based on emotional states to transition the cockpit from the current state to the target state. The control module is used to control at least one environmental control device in the cockpit to perform progressive linkage actions based on the cockpit linkage strategy.

[0095] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0096] The vehicle provided in this embodiment is used to execute the above-described cockpit linkage control method, and therefore can achieve the same effect as the above-described implementation method.

[0097] When using integrated units, the vehicle may include a processing module and a storage module. The processing module is used to control and manage the vehicle's actions. The storage module supports the vehicle in executing program code and data.

[0098] The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits as disclosed in this application. The processor may also be a combination of computing functions, such as a combination of one or more microprocessors, a combination of digital signal processing (DSP) and a microprocessor, etc., and the storage module may be a memory.

[0099] This embodiment also provides a computer-readable storage medium (including but not limited to disk storage, CD-ROM, optical storage, etc.) storing computer program code. When the computer program code is run on a computer, the computer executes the above-mentioned related method steps to implement the cockpit linkage control method provided in the above embodiment.

[0100] This embodiment also provides a computer program product. When the computer program product is run on a computer, it causes the computer to perform the above-mentioned related steps to realize the cockpit linkage control method provided in the above embodiment.

[0101] The beneficial effects of the above embodiments can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0102] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0103] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0104] In the description of this disclosure, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure.

[0105] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0106] The above are merely embodiments of this disclosure and are not intended to limit the scope of this disclosure. Various modifications and variations can be made to this disclosure by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the scope of the claims of this disclosure.

Claims

1. A cockpit linkage control method, characterized in that, Applied to vehicle control systems, the method includes: Acquire facial images of personnel inside the cockpit and determine the facial interference state of the corresponding facial images of the personnel; wherein, the facial interference state is determined by one or more of the following: lighting conditions, degree of facial occlusion, and posture changes; The facial image is input into multiple emotion detection models to obtain the corresponding output detection results; Based on the facial interference state, the evaluation weight corresponding to each emotion detection model is determined, and the detection results are weighted and fused according to the evaluation weight to obtain the emotional state of the person. Based on the emotional state, a cockpit linkage strategy is generated to transition the cockpit from the current state to the target state. Based on the cockpit linkage strategy, at least one environmental control device in the cockpit is controlled to perform progressive linkage actions.

2. The cockpit linkage control method according to claim 1, characterized in that, Based on the emotional state, a cockpit linkage strategy is generated to transition the cockpit from the current state to the target state, specifically including: Based on a preset sliding time window, the temporal change characteristics corresponding to the emotional state are determined; Based on the temporal change characteristics, an emotional intervention state is determined to transition the cabin from the current state to the target state. Obtain the target cabin environment parameters associated with the emotional intervention state, and generate a cabin linkage strategy based on the target cabin environment parameters.

3. The cockpit linkage control method according to claim 2, characterized in that, Based on the target cockpit environment parameters, a cockpit linkage strategy is generated, specifically including: Obtain the current parameter values ​​of each environmental control device in the cockpit; The control parameter sequence corresponding to the transition from the current parameter value to the target cabin environment parameter is calculated using a preset interpolation algorithm. Based on the sequence of control parameters, a corresponding cockpit linkage strategy is generated.

4. The cockpit linkage control method according to claim 3, characterized in that, Using a preset interpolation algorithm, the sequence of control parameters corresponding to the transition from the current parameter value to the target cabin environment parameter is calculated, specifically including: Based on the vehicle's driving status information, determine the driving risk index corresponding to the current vehicle; Based on the driving risk index, the transition time corresponding to the transition of the cabin from the current parameter value to the target cabin environment parameter is determined; wherein, the transition time and the driving risk index are negatively correlated. By using a preset interpolation algorithm, the control parameters corresponding to each time point in the transition duration are determined, so as to obtain the control parameter sequence when smoothly transitioning from the current parameter value to the target cabin environment parameter based on the control parameters.

5. The cockpit linkage control method according to claim 4, characterized in that, Before generating the corresponding cockpit linkage strategy based on the control parameter sequence, the method further includes: Determine whether the driving risk index meets the preset linkage intervention conditions; If not, the control parameter sequence is adjusted so that the intensity of the action corresponding to the adjusted control parameter sequence is adapted to the driving risk index.

6. The cockpit linkage control method according to claim 1, characterized in that, The detection result refers to an emotion probability vector composed of confidence scores corresponding to multiple emotion dimensions. The detection result is weighted and fused according to the evaluation weights to obtain the emotional state of the person, specifically including: Based on the evaluation weights, the emotion probability vectors are weighted and fused to obtain the corresponding emotion recognition vectors; Multimodal environmental parameters reflecting the emotional state of the person are collected, and the emotion recognition vector is corrected based on the multimodal environmental parameters to obtain the corrected emotion recognition vector. The emotion type corresponding to the emotion dimension with the highest confidence in the modified emotion recognition vector is taken as the emotional state of the person.

7. The cockpit linkage control method according to claim 1, characterized in that, After obtaining the emotional state of the person, the method further includes: The dominant emotion of the group is determined based on the emotional state and decision weight of each individual; wherein the decision weight is determined based on at least one of the individual's cabin location, age, and gender. By responding to the dominant emotions of the group, a cockpit linkage strategy corresponding to the cockpit is generated.

8. The cockpit linkage control method according to claim 7, characterized in that, By responding to the dominant emotion of the group, a corresponding cockpit linkage strategy is generated, specifically including: Based on the emotional state of the driver among the personnel, determine the corresponding emotional response type of the cockpit; Based on the emotional response type, a cockpit linkage strategy corresponding to the cockpit is generated in response to the driver's emotional state or the dominant group emotion.

9. A computer-readable storage medium storing computer-executable instructions, characterized in that, Applied to vehicle control systems, the computer-executable instructions are set as follows: The cockpit linkage control method as described in any one of claims 1-8.

10. A vehicle, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to: perform the cockpit linkage control method as described in any one of claims 1-8.

Images