Cabin mental health adjusting method and device and electronic equipment

Abstract

The invention discloses a cabin mental health adjustment method and device and electronic equipment, and the method comprises the steps: obtaining original data sorted based on a time sequence from multiple dimensions after a vehicle is started; extracting data features of the original data of the multiple dimensions to obtain a data feature vector corresponding to each modal; inputting the data feature vectors into a pre-trained multi-modal time sequence fusion model according to a time sequence to obtain a first recognition result; the first recognition result is used for representing the emotion category and / or emotion intensity of the person in the vehicle; determining at least one target adjustment strategy from an adjustment strategy library based on the first identification result and the attribute information of the person in the vehicle; and adjusting the equipment in the cabin based on the target adjustment strategy. The psychological state of the person in the vehicle can be adjusted in time when negative emotions such as anxiety and fatigue of the driver are found, the driving safety is improved, and the user experience is optimized. According to the invention, user health management is promoted, and vehicle intelligence can be enhanced.

Description

Technical Field

[0001] This application relates to the field of automotive technology, and in particular to a method, device, and electronic device for adjusting cabin mental health. Background Technology

[0002] With the development of modern transportation, people are spending more and more time in their cars, and the impact of the in-car environment on their mental health is receiving increasing attention. Traditional smart cockpit designs mainly focus on comfort and entertainment functions, with limited ability to regulate the mental health of in-car occupants.

[0003] In the automotive cabin environment, the psychological state of occupants is influenced by a variety of complex factors. Traditional psychological state recognition methods often rely on sensor data, such as simple facial expression recognition or a single physiological indicator (like heart rate), which cannot accurately identify the complex psychological states of occupants or effectively regulate their physical health. Summary of the Invention

[0004] This disclosure provides a method, apparatus, device, and storage medium for cabin mental health adjustment, in order to at least solve the above-mentioned technical problems existing in the prior art.

[0005] According to a first aspect of this application, a method for adjusting mental health in a cockpit is provided, comprising:

[0006] After the vehicle starts, raw data sorted by time is obtained from multiple dimensions;

[0007] Extract data features from the original data across multiple dimensions to obtain a data feature vector corresponding to each modality;

[0008] The multimodal data feature vectors are input into a pre-trained multimodal temporal fusion model in chronological order to obtain a first recognition result; the first recognition result is used to characterize the emotion category and / or emotion intensity of the occupants in the vehicle.

[0009] Based on the first identification result and the attribute information of the occupants, at least one target adjustment strategy is determined from the adjustment strategy library;

[0010] The in-cabin equipment is adjusted based on the target adjustment strategy.

[0011] In one possible implementation, the raw data includes at least visual data, voice data, and vehicle driving data; the acquisition of raw data sorted chronologically from multiple dimensions includes:

[0012] Visual data of occupants inside the vehicle is collected using an image sensor; the visual data includes facial expression data and body movement data.

[0013] Voice data of occupants is collected using a voice sensor;

[0014] Vehicle driving data is collected through vehicle sensors; the driving data includes at least one of vehicle speed, acceleration, steering angle, and braking force.

[0015] In one possible implementation, extracting data features from the original data across multiple dimensions to obtain a data feature vector corresponding to each modality includes:

[0016] The visual data is sequentially subjected to face detection, key point localization and limb detection, and time-series facial features and eye features are extracted to obtain a visual feature vector.

[0017] The speech data is denoised, and the prosodic features of the denoised speech data are extracted to obtain a speech feature vector.

[0018] The driving data is filtered, and the filtered driving data is calibrated based on the attribute information of the people in the vehicle to obtain the driving feature vector.

[0019] In one possible implementation, the step of inputting the multimodal data feature vectors into a pre-trained multimodal temporal fusion model in chronological order to obtain a first recognition result includes:

[0020] The data feature vectors of each modality are respectively input into the multimodal temporal fusion model based on the time order;

[0021] The fusion features output by the multimodal temporal fusion model are obtained; the fusion features are obtained by weighted fusion of the data feature vectors of each modality by the cross-modal attention mechanism in the multimodal temporal fusion model.

[0022] Based on the fusion features, a first recognition result is obtained.

[0023] In one possible implementation, the attribute information of the occupants includes at least one of the following:

[0024] Age, gender, driving habits, and driving habit patterns and historical psychological state data derived from historical data.

[0025] In one possible implementation, determining at least one target adjustment strategy from the adjustment strategy library based on the first identification result and the attribute information of the occupants includes:

[0026] By combining the first identification result with the age, gender, driving habits of the occupants, and at least one of the driving habit patterns and historical psychological state data obtained from historical data, a target execution strategy is obtained.

[0027] The target execution strategy includes at least one of the following:

[0028] Environmental adjustment strategy, audio playback strategy, and seat function strategy.

[0029] In one possible implementation, the target execution strategy includes adjustment levels;

[0030] Based on the aforementioned adjustment levels, a combination of adjustments for environmental adjustment strategies, audio playback strategies, and seat function strategies is determined.

[0031] In one possible implementation, it further includes:

[0032] After executing the target execution strategy, continuously acquire real-time data from multiple dimensions;

[0033] The temporal features of the real-time data are input into the multimodal temporal fusion model to obtain a second identification result; the second identification result is used to characterize the adjustment effect.

[0034] Obtain the second recognition result of the real-time data output by the multimodal temporal fusion model;

[0035] Calculate the deviation between the second identification result and the target's mental state;

[0036] Based on the deviation, update the target execution strategy.

[0037] According to a second aspect of this application, a cabin mental health adjustment device is provided, comprising:

[0038] The acquisition module is used to acquire raw data sorted by time from multiple dimensions after the vehicle is started.

[0039] The extraction module is used to extract data features from the original data in multiple dimensions to obtain a data feature vector corresponding to each modality.

[0040] The recognition module is used to input the multimodal data feature vectors into a pre-trained multimodal temporal fusion model in chronological order to obtain a first recognition result; the first recognition result is used to characterize the emotion category and / or emotion intensity of the occupants in the vehicle.

[0041] The determination module is used to determine at least one target adjustment strategy from the adjustment strategy library based on the first identification result and the attribute information of the occupants in the vehicle.

[0042] An adjustment module is used to adjust the equipment in the cockpit based on the target adjustment strategy.

[0043] According to a third aspect of this application, an electronic device is provided, comprising:

[0044] At least one processor; and

[0045] A memory communicatively connected to the at least one processor; wherein,

[0046] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in this application.

[0047] According to a fourth aspect of this application, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the methods described in this application.

[0048] According to a fifth aspect of this application, a computer program product is provided, comprising a computer program or instructions that, when executed by a processor, implement the method described in this application.

[0049] By utilizing the technical solution of this application, when negative emotions such as driver anxiety and fatigue are detected, the psychological state of the occupants in the vehicle can be adjusted in a timely manner, thereby improving driving safety and optimizing user experience. This application not only promotes user health management but also enhances vehicle intelligence.

[0050] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0051] The above and other objects, features, and advantages of exemplary embodiments of this application will become readily apparent from the following detailed description taken in conjunction with the accompanying drawings. Several embodiments of this application are illustrated in the drawings by way of example and not limitation, in which:

[0052] In the accompanying drawings, the same or corresponding reference numerals indicate the same or corresponding parts.

[0053] Figure 1 This paper illustrates the implementation flow of the cabin mental health adjustment method in an embodiment of this application. Figure 1 ;

[0054] Figure 2 This paper illustrates the implementation flow of the cabin mental health adjustment method in an embodiment of this application. Figure 2 ;

[0055] Figure 3 A schematic diagram of the cabin mental health adjustment device in an embodiment of this application is shown;

[0056] Figure 4A schematic diagram of the composition structure of an electronic device according to an embodiment of this application is shown. Detailed Implementation

[0057] To make the objectives, features, and advantages of this application more apparent and understandable, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0058] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0059] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0060] In the following description, the terms "first" and "second" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first" and "second" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0061] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0062] It should be understood that in the various embodiments of this application, the sequence number of each implementation process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0063] The following description, in conjunction with the accompanying drawings, introduces a method, device, and electronic device for adjusting cabin mental health provided in this application.

[0064] like Figure 1 As shown, this application provides a method for adjusting mental health in a cockpit, including:

[0065] S101: After the vehicle is started, raw data sorted by time order is obtained from multiple dimensions;

[0066] The cabin mental health adjustment method provided in this application can be applied to automobiles. Automobiles are equipped with various sensors, such as cameras, microphones, and vehicle sensors. This application can acquire raw data about occupants and the vehicle from multiple dimensions using these sensors. The raw data is arranged in chronological order.

[0067] It is understandable that after a car starts, the various sensors inside the vehicle are powered on and activated. Therefore, this application uses "vehicle start" as a trigger signal to drive the various sensors to acquire data on the occupants or the vehicle in various dimensions. Furthermore, this application enables the in-vehicle sensors to automatically collect data by starting the car, eliminating the need for manual user operation. This ensures the continuity of data collection across various dimensions and avoids interference with the user's driving / riding experience, aligning with the user's desire for a seamless smart car experience.

[0068] In some embodiments, the raw data includes at least visual data, voice data, and vehicle driving data; obtaining the raw data sorted chronologically from multiple dimensions includes:

[0069] Visual data of occupants inside the vehicle is collected using an image sensor; the visual data includes facial expression data and body movement data.

[0070] Voice data of occupants is collected using a voice sensor;

[0071] Vehicle driving data is collected through vehicle sensors; the driving data includes at least one of vehicle speed, acceleration, steering angle, and braking force.

[0072] In this application, the multiple dimensions can include visual dimensions, audio dimensions, and driving behavior dimensions. For the visual dimension, this application collects visual data of occupants in the vehicle using image sensors; the visual data includes facial expression data and body movement data. Visual data can include: facial expressions (degree of frowning, width of the eye opening, curvature of the corners of the mouth), body movements (stiffness of posture, whether the hands are gripping the steering wheel tightly), and eye movement trajectories (blinking frequency, fixation point). This application can install a high-definition camera in the cabin to collect visual data at a rate of more than 30 frames per second. Audio data can include: rapid conversations, complaints, tone of voice, speech rate, and emotional vocalizations (such as sighs, shouts), etc. It can be obtained by using microphones installed in the cabin. The driving behavior dimension can include driving speed, acceleration, braking frequency, steering angle, and throttle opening, etc. Driving data can be collected by vehicle sensors installed on the car. In addition, the dimension can also include physiological signal dimensions, such as heart rate, heart rate variability, skin conductance response, and respiratory rate, etc. Physiological data can be collected by physiological sensors installed in the seat. Dimensions can also include cabin environment dimensions, such as cabin temperature, light intensity, noise levels, and air quality. Physiological data can be collected through environmental sensors installed within the cabin. These sensors can include temperature and humidity sensors, as well as air quality sensors.

[0073] S102, extract data features from the original data in multiple dimensions to obtain a data feature vector corresponding to each modality;

[0074] In some embodiments, extracting data features from the original data across multiple dimensions to obtain a data feature vector corresponding to each modality includes:

[0075] The visual data is sequentially subjected to face detection, key point localization and limb detection, and time-series facial features and eye features are extracted to obtain a visual feature vector.

[0076] The speech data is denoised, and the prosodic features of the denoised speech data are extracted to obtain a speech feature vector.

[0077] The driving data is filtered, and the filtered driving data is calibrated based on the attribute information of the people in the vehicle to obtain the driving feature vector.

[0078] In this application, facial expression features of the face region can be extracted through face detection and key point localization. For limb detection, only key points of the upper body (such as shoulders, elbows, and hands) can be detected, focusing on "limb postures during driving" (such as the posture of gripping the steering wheel tightly when nervous), avoiding the computational consumption of full limb detection. Based on time-series image acquisition, temporal facial features and temporal eye features are obtained. This application concatenates "temporal facial features" (such as 20-dimensional features related to brow bone movement and corner of the mouth movement) with "temporal eye features" (such as 10-dimensional features related to blinking frequency and pupil changes) to form a fixed-dimensional visual feature vector (such as 30-dimensional), ensuring subsequent alignment and fusion with other modal vectors.

[0079] In speech data, prosodic features (intonation, speech rate, and rhythm) are core indicators reflecting emotion. For speech data, this application removes noise components from the raw speech data to distinguish between "segments with speech" and "segments without speech," and only extracts features from the "segments with speech" to avoid noise data in silent periods interfering with feature quality (e.g., wind noise in the absence of speech does not require feature extraction). Then, intonation, speech rate, and rhythm are extracted from the speech data to obtain speech feature vectors, and the dimensions of these speech features are strongly correlated with emotion.

[0080] In the driving data, Kalman filtering or moving average filtering is used to smooth the original driving data (such as 10Hz speed and acceleration). This application can also calibrate the filtered driving data based on the attribute information of the occupants. Specifically, a "personalized calibration model" is established based on the attribute information of the occupants (such as age, years of driving experience, and historical driving habits). For example, the normal driving behavior threshold of a "group of the same age and years of driving experience" is first statistically analyzed (e.g., the normal acceleration threshold of a 30-year-old group with 5 years of driving experience is <2.5m / s). 2 Then, the current filtered driving data is compared with this threshold to calculate the "deviation" (e.g., if the current acceleration is 2.3 m / s², the deviation = 2.3 / 2.5 = 0.92). The "deviation" is used as the core feature value (the higher the deviation, the more likely it is abnormal driving caused by emotions). The "calibrated speed fluctuation features, acceleration deviation features, braking frequency features, steering angle fluctuation features, etc." (e.g., each feature corresponds to 1-2 dimensions) are concatenated to form a driving feature vector (e.g., 8 dimensions) to ensure that the features can reflect the "correlation between personalized driving behavior and psychological state".

[0081] S103, the multimodal data feature vectors are input into a pre-trained multimodal temporal fusion model in chronological order to obtain a first recognition result; the first recognition result is used to characterize the emotion category and / or emotion intensity of the occupants in the vehicle;

[0082] In some embodiments, the step of inputting the multimodal data feature vectors into a pre-trained multimodal temporal fusion model in chronological order to obtain a first recognition result includes:

[0083] The data feature vectors of each modality are respectively input into the multimodal temporal fusion model based on the time order;

[0084] The fusion features output by the multimodal temporal fusion model are obtained; the fusion features are obtained by weighted fusion of the data feature vectors of each modality by the cross-modal attention mechanism in the multimodal temporal fusion model.

[0085] Based on the fusion features, a first recognition result is obtained.

[0086] In this application, the feature vectors of each modality are transformed into "temporal feature sequences" and input into a multimodal temporal fusion model. Specifically, visual, speech, and driving modalities are input into the multimodal temporal fusion model. The multimodal temporal fusion model receives parallel multimodal temporal sequences and performs joint analysis on the multimodal temporal features within the same time period. The multimodal temporal fusion model in this application can employ architectures such as Transformer's TimeSformer or MViT.

[0087] The multimodal temporal fusion model in this application jointly analyzes multimodal temporal features within the same time period. Specifically, the cross-modal attention mechanism first encodes the visual feature vector, speech feature vector, and driving feature vector, mapping each modal feature to the same semantic space. Then, similarity is calculated to obtain the attention score for each modality. The fused features for that time window are obtained by multiplying each modal feature vector by its corresponding attention weight and summing the results. Repeat the weighting process described above for all time windows to obtain the temporal fusion feature sequence for the entire 1-minute period. This sequence contains both the fusion features of each time period and retains the trend of feature changes over time. Then, calculate the global fusion feature of the temporal fusion feature sequence. Input the global fusion feature into the classification head of the multimodal temporal fusion model to output the probability distribution of each psychological state category. For example, output "calm: 0.1, anxiety: 0.85, fatigue: 0.05". Select the category with the highest probability as the first identification result, i.e., "anxiety" (if the probability exceeds a preset threshold, such as 0.6, then output this result; if it does not reach the threshold, it can be marked as "pending confirmation" to avoid low-confidence misjudgment).

[0088] This application utilizes a pre-trained multimodal temporal fusion model, eliminating the need for training from scratch and requiring only fine-tuning with cockpit scene data, thus reducing development costs and shortening the deployment cycle. This application constructs a multimodal psychological state recognition logic adapted to intelligent cockpit scenarios. This enables the multimodal temporal fusion model to dynamically utilize reliable information from each modality and capture the temporal trends of psychological state changes, ultimately outputting accurate and reliable first-order recognition results (such as anxiety, calmness, etc.).

[0089] S104, based on the first identification result and the attribute information of the occupants, determine at least one target adjustment strategy from the adjustment strategy library;

[0090] In some embodiments, the attribute information of the occupants includes at least one of the following:

[0091] Age, gender, driving habits, and driving habit patterns and historical psychological state data derived from historical data.

[0092] In this application, by using age, gender, driving habits, and driving habit patterns and historical psychological state data obtained from historical data, the general recognition bias of the multimodal model can be corrected so that the adjustment strategy can be tailored to the user's specific needs (such as different strategies for experienced drivers and novice drivers).

[0093] This application can adjust the first recognition result obtained by the multimodal temporal fusion model by using the attribute information of the occupants in the vehicle. For example, the multimodal temporal fusion model may misjudge "slight frowning of an elderly driver" as "calm" (because the facial skin of the elderly is loose, and the frowning amplitude is smaller than that of young people). However, by combining the "age attribute (60 years old)", the model can adjust the "judgment threshold of the frowning feature" (a frowning amplitude >0.6 in young people is judged as anxiety, and >0.4 in the elderly is judged as anxiety), thus avoiding missed judgments.

[0094] In some embodiments, determining at least one target adjustment strategy from the adjustment strategy library based on the first identification result and the attribute information of the occupants includes:

[0095] By combining the first identification result with the age, gender, driving habits of the occupants, and at least one of the driving habit patterns and historical psychological state data obtained from historical data, a target execution strategy is obtained.

[0096] The target execution strategy includes at least one of the following:

[0097] Environmental adjustment strategy, audio playback strategy, and seat function strategy.

[0098] Specifically, in this application, one to three environmental adjustment strategies, audio playback strategies, and seat function strategies can be selected and combined (to avoid interference caused by too many strategies) to form the final "target execution strategy". The combination logic follows the principles of "complementary synergy and no interference".

[0099] For example, if the user is a "25-year-old female, a new driver, whose initial identification result is anxiety, and whose historical data shows a preference for aromatherapy and music," the final combination would be "ambient adjustment (light blue cool light and citrus aromatherapy) and audio playback (low-volume soothing piano music)" (excluding full-body massage to avoid distraction). Or, if the user is a "55-year-old male, an experienced driver, whose initial identification result is fatigue, and whose historical data shows sensitivity to massage," the final combination would be "ambient adjustment (soft bright light) and audio playback (upbeat folk music)" (excluding massage, retaining energizing strategies).

[0100] In this application, the target execution strategy focuses on three dimensions that are strongly related to the cabin experience: "environment, audio, and seating". Each type of strategy has a clear "adjustment goal" and "scenario adaptation method" to ensure that it can effectively alleviate the corresponding psychological state.

[0101] In some embodiments, the target execution strategy includes adjustment levels;

[0102] Based on the aforementioned adjustment levels, a combination of adjustments for environmental adjustment strategies, audio playback strategies, and seat function strategies is determined.

[0103] In this application, the adjustment level is based on the severity of the psychological state and the user's tolerance, and the adjustment strategy's 'intensity, scope and priority' is quantitatively graded. The division needs to be combined with the 'quantitative indicators of the first identification result' and the 'in-vehicle personnel attribute information' to ensure that the level reflects both the 'emotional urgency' and the 'user acceptance'.

[0104] The adjustment level of this application is usually divided into three levels: mild, moderate, and severe.

[0105] Among them, mild adjustment focuses on "minor intervention without interfering with driving," with low adjustment intensity and narrow range, only alleviating initial emotions through basic strategies. For example, the first identification result is "mild anxiety / mild fatigue," with quantitative indicators in the low range (such as anxiety level score of 30-50 points, fatigue index of 40-60 points); the user is a novice driver (low tolerance) or in a complex driving scenario (such as traffic jam, highway).

[0106] Moderate regulation focuses on "effective intervention and balancing effects and interference," with moderate regulation intensity and a suitable range. Combining 2-3 types of strategies enhances the relief effect. For example, the first identification result is "moderate anxiety / moderate fatigue," with quantitative indicators in the middle range (such as anxiety level score of 50-70 points and fatigue index of 60-80 points); the user is an experienced driver (with high tolerance) or in a stable driving scenario (such as urban expressways or suburban roads).

[0107] Intensive regulation focuses on "powerful intervention and prioritizing emotional relief," characterized by high intensity and broad scope, combining multi-dimensional strategies, and adding proactive reminders when necessary. For example, the first identification result is "severe anxiety / severe fatigue," with quantitative indicators in the high range (e.g., anxiety level score of 70-100 points, fatigue index of 80-100 points); the user is in a safe driving scenario (e.g., an open road) and has no history of strategy allergies (e.g., tolerance to strong massage).

[0108] S105, Adjust the in-cabin equipment based on the target adjustment strategy.

[0109] In this application, the in-cabin equipment includes a cabin lighting system, a cabin audio system, and a seating system. The lighting system typically includes "driver's area lights, center console ambient lights, and rear ambient lights," and adjustments must balance driver visual safety and passenger experience. Priority is given to adjusting "lights in areas outside the driver's field of vision" (such as center console ambient lights and rear lights), with only minor brightness adjustments to the driver's dashboard lights (to avoid affecting dashboard readings). Color temperature / brightness adjustments use a "gradual transition" (e.g., color temperature from 3000K to 5000K takes 3 seconds, brightness from 100% to 40% takes 2 seconds) to avoid sudden changes in light that could cause eye strain due to pupil constriction / dilation. If "night driving" is detected, the maximum brightness adjustment is set to 50% (to avoid strong light affecting the visibility of oncoming vehicles); if "daytime driving" is detected, the brightness can be increased to 80% (to ensure an ambiance).

[0110] For the audio system, a "smooth gain" approach is adopted (e.g., the gain change from 10% to 15% should be completed in 1 second to avoid a sudden increase in volume that might startle the user), and the maximum volume should not exceed 30% (lower than the navigation voice volume to ensure that navigation instructions can be heard clearly). When switching tracks, if navigation voice is currently playing, the switch should be delayed until the voice ends (to avoid interrupting navigation). If the user manually switches music or turns off the audio, the system should record "user feedback" and prioritize excluding that track or lowering the priority of the audio strategy in subsequent similar strategies.

[0111] For the seating system, the massage function should prioritize "light local massage" (such as for the lower back and back), and only activate "full-body massage" when making heavy adjustments. The massage intensity should be gradually increased to avoid sudden heavy massage that could cause the body to tilt. The seat angle (backrest and headrest) adjustment range should be controlled within ±3-5°, and the adjustment process should be slow (1 second per degree) to ensure the driver can adapt to changes in posture in real time. If "the driver is turning, braking, or accelerating," immediately pause the seat adjustment and resume it after the operation is completed (to avoid the adjustment affecting steering wheel control or pedal operation).

[0112] In some embodiments, such as Figure 2 As shown, the cabin mental health adjustment method provided in this application also includes:

[0113] S201, after executing the target execution strategy, continuously acquire real-time data from multiple dimensions;

[0114] S202, the temporal features of the real-time data are input into the multimodal temporal fusion model to obtain a second identification result; the second identification result is used to characterize the adjustment effect;

[0115] S203, obtain the second recognition result of the real-time data recognition output by the multimodal time-series fusion model;

[0116] S204, Calculate the deviation between the second identification result and the target's psychological state;

[0117] S205, based on the deviation, update the target execution strategy.

[0118] In this application, after the target execution strategy is initiated (such as turning on a light blue light and soothing music), the logic of "multi-dimensional temporal data acquisition" must continue to acquire real-time data that is strongly correlated with the psychological state, ensuring that the "trend of emotional change after adjustment" can be captured. The temporal feature extraction method of the real-time data is consistent with that before adjustment (such as visual extraction of facial temporal features and speech extraction of prosodic temporal features), ensuring that the model input format is uniform, the recognition results are comparable, and avoiding "deviation of recognition results before and after" due to changes in feature extraction methods.

[0119] In this application, the target psychological state is the system's preset emotion improvement target (such as "the target of the anxiety state is to reduce to mild anxiety ≤50 points" and "the target of the fatigue state is to reduce to mild fatigue ≤40 points"). The system quantitatively judges whether the adjustment effect has been achieved by calculating the "deviation value between the second identification result and the target state".

[0120] The deviation value is defined as: |Quantitative value of the second identification result - Quantitative value of the target psychological state|. This application pre-sets a "deviation threshold" (e.g., 10 points) to determine whether the target has been met: If the deviation value is ≤10 points: the adjustment has met the target (e.g., a deviation of 8 points indicates the emotion is close to the target state), and the current strategy is maintained; if the deviation value is greater than 10 points: the adjustment has not met the target (e.g., a second identification result of 58 points indicates a deviation of 8 points → met the target; a result of 62 points indicates a deviation of 12 points → not met the target), and strategy optimization needs to be initiated. Based on the magnitude and direction of the deviation value (e.g., no improvement in emotion, insufficient improvement, or excessive improvement), a "targeted optimization plan" is selected from the adjustment strategy library, and the target execution strategy is updated to ensure that subsequent adjustments can reduce the deviation until the target is met.

[0121] The cabin mental health adjustment method provided in this application ensures the reliability of mental state recognition through multimodal fusion and refined processing, and can provide users with personalized and safe adjustment services, thereby improving user experience.

[0122] like Figure 3 As shown, this application provides a cabin mental health adjustment device, comprising:

[0123] The acquisition module 301 is used to acquire raw data sorted by time order from multiple dimensions after the vehicle is started.

[0124] Extraction module 302 is used to extract data features of the original data in multiple dimensions to obtain a data feature vector corresponding to each modality;

[0125] The recognition module 303 is used to input the multimodal data feature vectors into a pre-trained multimodal temporal fusion model in chronological order to obtain a first recognition result; the first recognition result is used to characterize the emotion category and / or emotion intensity of the occupants in the vehicle.

[0126] The determining module 304 is used to determine at least one target adjustment strategy from the adjustment strategy library based on the first identification result and the attribute information of the occupants in the vehicle.

[0127] The adjustment module 305 is used to adjust the equipment in the cockpit based on the target adjustment strategy.

[0128] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0129] The electronic device includes at least one processor and a memory communicatively connected to the at least one processor. The memory stores instructions executable by the at least one processor, which, when executed, enable the at least one processor to perform the cabin mental health adjustment method described in this application. The computer instructions are used to cause the computer to perform the cabin mental health adjustment method described in this application.

[0130] This application also provides a computer program product, including a computer program / instructions, which, when executed by a processor, implement the cabin mental health adjustment method of this application.

[0131] Figure 4 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0132] like Figure 4 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 802 or a computer program loaded from storage unit 808 into random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. The computing unit 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804.

[0133] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0134] The computing unit 801 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the cabin mental health adjustment method. For example, in some embodiments, the cabin mental health adjustment method can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the cabin mental health adjustment method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the cabin mental health adjustment method by any other suitable means (e.g., by means of firmware).

[0135] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0136] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0137] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0138] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0139] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0140] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0141] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for adjusting mental health in a cockpit, characterized in that, include: After the vehicle starts, raw data sorted by time is obtained from multiple dimensions; Extract data features from the original data across multiple dimensions to obtain a data feature vector corresponding to each modality; The multimodal data feature vectors are input into a pre-trained multimodal temporal fusion model in chronological order to obtain a first recognition result; the first recognition result is used to characterize the emotion category and / or emotion intensity of the occupants in the vehicle. Based on the first identification result and the attribute information of the occupants, at least one target adjustment strategy is determined from the adjustment strategy library; The in-cabin equipment is adjusted based on the target adjustment strategy.

2. The method according to claim 1, characterized in that, The raw data includes at least visual data, voice data, and vehicle driving data; the acquisition of raw data sorted chronologically from multiple dimensions includes: Visual data of occupants inside the vehicle is collected using an image sensor; the visual data includes facial expression data and body movement data. Voice data of occupants is collected using a voice sensor; Vehicle driving data is collected through vehicle sensors; the driving data includes at least one of vehicle speed, acceleration, steering angle, and braking force.

3. The method according to claim 2, characterized in that, The step of extracting data features from the original data across multiple dimensions to obtain a data feature vector corresponding to each modality includes: The visual data is sequentially subjected to face detection, key point localization and limb detection, and time-series facial features and eye features are extracted to obtain a visual feature vector. The speech data is denoised, and the prosodic features of the denoised speech data are extracted to obtain a speech feature vector. The driving data is filtered, and the filtered driving data is calibrated based on the attribute information of the people in the vehicle to obtain the driving feature vector.

4. The method according to claim 1, characterized in that, The step of inputting the multimodal data feature vectors into a pre-trained multimodal temporal fusion model in chronological order to obtain a first recognition result includes: The data feature vectors of each modality are respectively input into the multimodal temporal fusion model based on the time order; The fusion features output by the multimodal temporal fusion model are obtained; the fusion features are obtained by weighted fusion of the data feature vectors of each modality by the cross-modal attention mechanism in the multimodal temporal fusion model. Based on the fusion features, a first recognition result is obtained.

5. The method according to claim 1, characterized in that, The attribute information of the occupants in the vehicle includes at least one of the following: Age, gender, driving habits, and driving habit patterns and historical psychological state data derived from historical data.

6. The method according to claim 5, characterized in that, Based on the first identification result and the attribute information of the occupants, determining at least one target adjustment strategy from the adjustment strategy library includes: By combining the first identification result with the age, gender, driving habits of the occupants, and at least one of the driving habit patterns and historical psychological state data obtained from historical data, a target execution strategy is obtained. The target execution strategy includes at least one of the following: Environmental adjustment strategy, audio playback strategy, and seat function strategy.

7. The method according to claim 6, characterized in that, The target execution strategy includes adjustment levels; Based on the aforementioned adjustment levels, a combination of adjustments for environmental adjustment strategies, audio playback strategies, and seat function strategies is determined.

8. The method according to claim 1, characterized in that, Also includes: After executing the target execution strategy, continuously acquire real-time data from multiple dimensions; The temporal features of the real-time data are input into the multimodal temporal fusion model to obtain a second identification result; the second identification result is used to characterize the adjustment effect. Obtain the second recognition result of the real-time data output by the multimodal temporal fusion model; Calculate the deviation between the second identification result and the target's mental state; Based on the deviation, update the target execution strategy.

9. A cabin mental health adjustment device, characterized in that, include: The acquisition module is used to acquire raw data sorted by time from multiple dimensions after the vehicle is started. The extraction module is used to extract data features from the original data in multiple dimensions to obtain a data feature vector corresponding to each modality. The recognition module is used to input the multimodal data feature vectors into a pre-trained multimodal temporal fusion model in chronological order to obtain a first recognition result; the first recognition result is used to characterize the emotion category and / or emotion intensity of the occupants in the vehicle. The determination module is used to determine at least one target adjustment strategy from the adjustment strategy library based on the first identification result and the attribute information of the occupants in the vehicle. An adjustment module is used to adjust the equipment in the cockpit based on the target adjustment strategy.

10. An electronic device, characterized in that, At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

Images