Vehicle control method based on multi-mode perception, vehicle and storage medium
By acquiring data and quantifying the risk level of motion sickness through a multimodal perception module, the vehicle is controlled to implement anti-motion sickness strategies, which solves the problem of insufficient motion sickness perception in existing technologies and realizes dynamic intervention and personalized comfort enhancement.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-26
- Publication Date
- 2026-03-27
AI Technical Summary
Existing technologies lack the ability to proactively detect motion sickness, making it impossible to dynamically identify and actively intervene. Furthermore, existing individualized relief methods have limited effectiveness or are accompanied by side effects.
The system uses a multimodal perception module to acquire vehicle motion data, passenger status data, and in-vehicle environment data. It quantifies the risk level of motion sickness through a risk assessment model and comprehensively adjusts the air conditioning, fresh air device, and in-vehicle fragrance generator based on the risk level to implement an anti-motion sickness control strategy.
It enables proactive perception and quantitative assessment of motion sickness risk, allowing for precise intervention before passengers experience significant discomfort, thereby improving ride comfort and personalizing the travel experience.
Smart Images
Figure CN121734278A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of vehicle control technology, specifically providing a vehicle control method, vehicle, and storage medium based on multimodal perception. Background Technology
[0002] Motion sickness (commonly known as kinetosis) is a common physiological discomfort among RV passengers, especially those in the rest area at the rear of the vehicle. Its underlying mechanism stems from sensory conflict: when the vehicle is accelerating, decelerating, turning, or experiencing bumps, the inner ear's vestibular system senses the body's movement, while the visual system receives information about the relatively stationary interior environment. This perceptual inconsistency leads to a disruption in the brain's information integration, resulting in symptoms such as dizziness, nausea, and vomiting.
[0003] Currently, RV control systems primarily rely on basic temperature and humidity control within the cabin to address motion sickness. Airflow patterns are mostly fixed or only support simple directional airflow, failing to dynamically identify and proactively intervene in motion sickness risks. Personalized relief methods such as medication and fitness trackers generally suffer from limited effectiveness, poor user experience, or side effects. Overall, existing technologies still have significant shortcomings in addressing motion sickness caused by vehicle movement, mainly in the lack of forward-looking perception capabilities, precise airflow control, proactive intervention mechanisms, and multi-sensory collaborative strategies.
[0004] Accordingly, a new vehicle control scheme is needed in this field to solve the above problems. Summary of the Invention
[0005] In order to overcome the above-mentioned deficiencies, this application is made to provide a solution or at least a partial solution to the technical problem of the lack of forward-looking motion sickness perception capabilities in the prior art, and how to intervene in motion sickness through dynamic regulation.
[0006] In a first aspect, this application provides a vehicle control method based on multimodal perception, the method being applied to a vehicle equipped with a multimodal perception module; the method includes: acquiring multimodal perception data based on the multimodal perception module, wherein the multimodal perception data includes at least vehicle motion data, state data of at least one passenger, and in-vehicle environment data; acquiring the motion sickness risk level of the vehicle based on the multimodal perception data; and controlling the vehicle to execute a corresponding anti-motion sickness control strategy based on the motion sickness risk level.
[0007] In one technical solution of the above-mentioned vehicle control method based on multimodal perception, the multimodal perception module includes a vehicle motion perception unit, a visual attention sensor, and an environment perception unit; the acquisition of multimodal perception data includes: acquiring vehicle motion data based on the vehicle motion perception unit, the vehicle motion data including the vehicle's longitudinal acceleration, lateral acceleration, and yaw rate; acquiring state data of at least one passenger based on the visual attention sensor, the state data including at least the visual attention direction; and acquiring in-vehicle environment data based on the environment perception unit, the in-vehicle environment data including at least the in-vehicle carbon dioxide concentration.
[0008] In one technical solution of the above-mentioned vehicle control method based on multimodal perception, the multimodal perception module further includes a heart rate sensor; the acquisition of multimodal perception data further includes: acquiring the status data of at least one passenger based on the heart rate sensor, the status data including the heart rate variability data of at least one passenger.
[0009] In one technical solution of the above-mentioned vehicle control method based on multimodal perception, the step of controlling the vehicle to execute a corresponding anti-motion sickness control strategy based on the motion sickness risk level includes: controlling at least one of the vehicle's air conditioning device, fresh air device, and in-vehicle fragrance generator based on the motion sickness risk level, so as to control the vehicle to execute the corresponding anti-motion sickness control strategy.
[0010] In one technical solution of the above-mentioned vehicle control method based on multimodal perception, after controlling the vehicle to execute the corresponding anti-motion sickness control strategy based on the motion sickness risk level, the method further includes: continuously acquiring the multimodal perception data, and acquiring the updated motion sickness risk level based on the multimodal perception data; and controlling the vehicle to execute the corresponding anti-motion sickness control strategy according to the updated motion sickness risk level.
[0011] In one technical solution of the above-mentioned vehicle control method based on multimodal perception, obtaining the motion sickness risk level of the vehicle includes: determining the motion intensity index of the vehicle based on the vehicle motion data; and determining the motion sickness risk level based on the motion intensity index, the state data, and the in-vehicle environment data.
[0012] In one technical solution of the above-mentioned vehicle control method based on multimodal perception, the vehicle motion data includes the vehicle's longitudinal acceleration, lateral acceleration, and yaw rate; determining the vehicle's motion intensity index based on the vehicle motion data includes: determining the vehicle's motion intensity index based on the vehicle's longitudinal acceleration, lateral acceleration, and yaw rate using the following formula:
[0013]
[0014] In the formula, This is the exercise intensity index. , , To preset normalized weights, For longitudinal acceleration, For lateral acceleration, The yaw rate is angular velocity; and / or, determining the motion sickness risk level based on the motion intensity index, the state data, and the in-vehicle environment data includes: inputting the motion intensity index, the state data, and the in-vehicle environment data into a pre-trained risk assessment model, and outputting a real-time motion sickness risk index; determining the motion sickness risk level based on the real-time motion sickness risk index and a preset threshold range.
[0015] In one technical solution of the aforementioned vehicle control method based on multimodal perception, the motion sickness risk level includes low risk, medium risk, or high risk; controlling the vehicle to execute a corresponding anti-motion sickness control strategy includes: when the motion sickness risk level is low, controlling the vehicle to execute a basic comfort strategy, the basic comfort strategy including: controlling the air conditioning vents to be in a preset default position, and / or controlling the air conditioning to execute automatic air sweeping based on a preset sweeping mode; or, when the motion sickness risk level is medium, controlling the vehicle to execute a dynamic airflow cuing strategy, the dynamic airflow cuing strategy including: real-time control of the airflow direction of the air conditioning vents relative to the inertia generated by the vehicle's current movement. The direction of the airflow forms a preset angle, and / or the air conditioning's air delivery mode is switched to a preset random turbulence mode, which simulates the unpredictable breeze in nature; or, when the motion sickness risk level is high, the vehicle is controlled to execute a strong intervention and sensory reset strategy, which includes at least one of the following: controlling the airflow direction of the air conditioning vents towards the face or chest area of the at least one passenger, and adjusting the air conditioning temperature to a preset temperature and adjusting the air conditioning fan speed to a preset fan speed; controlling the in-vehicle fragrance generator to release a preset type of refreshing fragrance and maintaining it for a first preset time; controlling the vehicle's fresh air device to operate at maximum airflow for a second preset time.
[0016] In one technical solution of the above-mentioned vehicle control method based on multimodal perception, after controlling the vehicle to execute the corresponding anti-motion sickness control strategy, the method further includes: recording data during the execution of the anti-motion sickness control strategy, wherein the data includes at least multimodal perception data, motion intensity index, real-time motion sickness risk index, motion sickness risk level, and anti-motion sickness control strategy; and optimizing the risk assessment model or the anti-motion sickness control strategy based on the recorded data.
[0017] In a second aspect, this application provides a vehicle including a processor and a memory, the memory being adapted to store a plurality of program codes, the program codes being adapted to be loaded and executed by the processor to perform the vehicle control method based on multimodal perception described in any of the above-described technical solutions of the vehicle control method based on multimodal perception.
[0018] In a third aspect, a computer-readable storage medium is provided, wherein a plurality of program codes are stored therein, the program codes being adapted to be loaded and run by a processor to perform the vehicle control method based on multimodal perception described in any of the above-described technical solutions for the vehicle control method based on multimodal perception.
[0019] The above-described technical solutions of this application have at least one or more of the following beneficial effects:
[0020] This application discloses a vehicle control method based on multimodal perception, which is applied to a vehicle equipped with a multimodal perception module. The method includes: acquiring multimodal perception data based on the multimodal perception module, wherein the multimodal perception data includes at least vehicle motion data, state data of at least one passenger, and in-vehicle environment data; acquiring the motion sickness risk level of the vehicle based on the multimodal perception data; and controlling the vehicle to execute a corresponding anti-motion sickness control strategy based on the motion sickness risk level.
[0021] This application achieves proactive perception and quantitative assessment of motion sickness risk levels by multimodal fusion perception of vehicle motion data, passenger status data, and in-vehicle environment data. Based on the real-time determined motion sickness risk level, the application can control the vehicle to implement corresponding anti-motion sickness control strategies before passengers feel obvious discomfort, effectively improving ride comfort and enhancing the riding experience.
[0022] Furthermore, the step of controlling the vehicle to execute a corresponding anti-motion sickness control strategy based on the motion sickness risk level includes: controlling at least one of the vehicle's air conditioning unit, fresh air unit, and in-vehicle fragrance generator based on the motion sickness risk level, so as to control the vehicle to execute the corresponding anti-motion sickness control strategy.
[0023] This application utilizes a multi-dimensional integrated adjustment system, incorporating air conditioning, fresh air systems, and an in-vehicle fragrance generator, to prevent and alleviate motion sickness through multi-sensory synergy.
[0024] Furthermore, when the motion sickness risk level is medium, the vehicle is controlled to execute a dynamic airflow cuing strategy. This dynamic airflow cuing strategy includes: real-time control of the airflow direction of the air conditioning vents to form a preset angle with the direction of the inertial force generated by the vehicle's current movement, and / or switching the air conditioning's airflow mode to a preset random turbulence mode, which simulates the unpredictable breeze in nature; or, when the motion sickness risk level is high, the vehicle is controlled to execute a strong intervention and sensory reset strategy. This strong intervention and sensory reset strategy includes at least one of the following: controlling the airflow direction of the air conditioning vents towards the face or chest area of at least one passenger, and adjusting the air conditioning temperature to a preset temperature and adjusting the air conditioning fan speed to a preset fan speed; controlling the in-vehicle fragrance generator to release a preset type of refreshing fragrance and maintaining it for a first preset time; and controlling the vehicle's fresh air system to operate at maximum airflow for a second preset time.
[0025] This application can control the airflow direction of the air conditioning vents to adjust the airflow direction to create a specific angle with the direction of the vehicle's motion inertia, or to provide targeted airflow to the face or chest area for passengers at high risk of motion sickness, so that the intervention measures can be precisely matched with the current vehicle motion scenario and the individual needs of passengers, greatly improving comfort.
[0026] Furthermore, the method also includes: recording data during the execution of the motion sickness control strategy, wherein the data includes at least multimodal perception data, motion intensity index, real-time motion sickness risk index, motion sickness risk level, and motion sickness control strategy; and optimizing the risk assessment model or the motion sickness control strategy based on the recorded data.
[0027] This application, by recording and analyzing data during the execution of motion sickness control strategies, can continuously optimize risk assessment models and motion sickness control strategies, adaptively making risk assessment more accurate and control interventions more tailored to passengers' individual reactions and needs, thereby achieving long-term personalized comfort improvement. Attached Figure Description
[0028] The preferred embodiments of this application are described below with reference to the accompanying drawings, in which:
[0029] Figure 1 This is a schematic flowchart of the main steps of a vehicle control method based on multimodal perception according to an embodiment of this application;
[0030] Figure 2 This is a detailed flowchart illustrating the steps of a vehicle control method based on multimodal perception according to an embodiment of this application.
[0031] Figure 3This is a schematic diagram of the structure of a vehicle control system based on multimodal perception according to an embodiment of this application;
[0032] Figure 4 This is a schematic diagram of the main structure of a vehicle according to an embodiment of this application.
[0033] List of reference numerals in the attached diagram:
[0034] 11: Memory; 12: Processor. Detailed Implementation
[0035] Some embodiments of this application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely illustrative of the technical principles of this application and are not intended to limit the scope of protection of this application.
[0036] In the description of this application, "module" and "processor" can include hardware, software, or a combination of both. A module can include hardware circuitry, various suitable sensors, communication ports, memory, and can also include software components, such as program code, or a combination of software and hardware. A processor can be a central processing unit, microprocessor, image processor, digital signal processor, or any other suitable processor. The processor has data and / or signal processing capabilities. The processor can be implemented in software, in hardware, or a combination of both. Non-transitory computer-readable storage media includes any suitable medium capable of storing program code, such as magnetic disks, hard disks, optical disks, flash memory, read-only memory, random access memory, etc. The term "A and / or B" means all possible combinations of A and B, such as only A, only B, or A and B. The terms "at least one A or B" or "at least one of A and B" have a similar meaning to "A and / or B" and can include only A, only B, or A and B. The singular terms "a" or "this" can also include plural forms.
[0037] Currently, RV control systems primarily rely on basic temperature and humidity control within the cabin to address motion sickness. Airflow patterns are mostly fixed or only support simple directional airflow, failing to dynamically identify and proactively intervene in motion sickness risks. Personalized relief methods such as medication and fitness trackers generally suffer from limited effectiveness, poor user experience, or side effects. Overall, existing technologies still have significant shortcomings in addressing motion sickness caused by vehicle movement, mainly in the lack of forward-looking perception capabilities, precise airflow control, proactive intervention mechanisms, and multi-sensory collaborative strategies.
[0038] To address this, this application provides a vehicle control method based on multimodal perception. The method is applied to a vehicle equipped with a multimodal perception module. The method includes: acquiring multimodal perception data based on the multimodal perception module, wherein the multimodal perception data includes at least vehicle motion data, the status data of at least one passenger, and in-vehicle environmental data; acquiring the motion sickness risk level of the vehicle based on the multimodal perception data; and controlling the vehicle to execute a corresponding anti-motion sickness control strategy based on the motion sickness risk level. This application achieves proactive perception and quantitative assessment of motion sickness risk level by performing multimodal fusion perception of vehicle motion data, passenger status data, and in-vehicle environmental data. Based on the real-time determined motion sickness risk level, it can control the vehicle to execute corresponding anti-motion sickness control strategies before passengers experience significant discomfort, effectively improving ride comfort and enhancing the passenger experience.
[0039] See appendix Figure 1 , Figure 1 This is a schematic flowchart illustrating the main steps of a vehicle control method based on multimodal perception according to an embodiment of this application. Figure 1 As shown in the embodiment of this application, the vehicle control method based on multimodal perception is applied to a vehicle, and the vehicle is equipped with a multimodal perception module; the method mainly includes the following steps S101-S103.
[0040] Step S101: Based on the multimodal perception module, acquire multimodal perception data, wherein the multimodal perception data includes at least vehicle motion data, status data of at least one passenger, and in-vehicle environment data.
[0041] Step S102: Based on the multimodal perception data, obtain the motion sickness risk level of the vehicle.
[0042] Step S103: Based on the motion sickness risk level, control the vehicle to execute the corresponding motion sickness control strategy.
[0043] Based on the above steps S101-S103, this application achieves a forward-looking quantitative assessment of motion sickness risk level by performing multimodal fusion perception of vehicle motion data, passenger status data and in-vehicle environment data. Based on the real-time determined motion sickness risk level, it can proactively control the vehicle to implement corresponding anti-motion sickness control strategies before passengers feel obvious discomfort, thereby effectively improving riding comfort and enhancing the riding experience.
[0044] The following is in conjunction with the appendix Figure 2 The following will provide further explanation of steps S101-S103.
[0045] Regarding step S101, in one embodiment, the multimodal perception module includes a vehicle motion perception unit, a visual attention sensor, and an environment perception unit; acquiring multimodal perception data includes: acquiring vehicle motion data based on the vehicle motion perception unit, the vehicle motion data including the vehicle's longitudinal acceleration, lateral acceleration, and yaw rate; acquiring state data of at least one passenger based on the visual attention sensor, the state data including at least the visual attention direction; and acquiring in-vehicle environment data based on the environment perception unit, the in-vehicle environment data including at least the in-vehicle carbon dioxide concentration.
[0046] Specifically, the multimodal perception module includes a vehicle motion perception unit, a visual attention sensor, and an environmental perception unit. The vehicle motion perception unit can be an inertial measurement unit (IMU) used to collect vehicle motion data reflecting the vehicle's dynamic characteristics, including longitudinal acceleration, lateral acceleration, and yaw rate. Optionally, the vehicle motion perception unit can also be connected to the vehicle's CAN bus to synchronously acquire additional vehicle operation data such as vehicle speed, turn signal status, and gear position signals, thereby providing a more comprehensive and accurate reflection of the vehicle's real-time motion and control status.
[0047] The visual attention sensor includes cameras installed inside the vehicle. Using pre-set computer vision algorithms, it identifies and outputs the visual attention direction of at least one passenger, for example, determining whether the passenger's gaze is primarily directed towards the side window, front window, or in-vehicle screen. It should be noted that the image acquisition and processing only analyzes the passenger's gaze direction and does not perform facial recognition, identity verification, or continuous video recording, thus fully protecting passenger privacy.
[0048] The environmental sensing unit includes gas sensors, such as a carbon dioxide sensor, to monitor the carbon dioxide concentration inside the vehicle and assess the freshness of the air inside the vehicle. Optionally, the environmental sensing unit may also include temperature and humidity sensors to simultaneously acquire temperature and humidity data inside the vehicle, thereby providing a more comprehensive input for the overall assessment of the in-vehicle environment.
[0049] Through the aforementioned multi-dimensional perception methods, three key types of information can be acquired simultaneously: vehicle motion data, passenger status data, and in-vehicle environment data, providing a comprehensive and real-time data foundation for subsequent motion sickness risk assessment.
[0050] In one embodiment, the multimodal sensing module further includes a heart rate sensor; the acquisition of multimodal sensing data further includes: acquiring state data of at least one passenger based on the heart rate sensor, the state data including heart rate variability data of at least one passenger.
[0051] Specifically, the multimodal perception module also includes a heart rate sensor, which can be implemented using non-contact measurement methods, such as remote photoplethysmography (rPPG) based on cameras. By analyzing subtle skin color changes in the acquired video signals, the heart rate variability data of at least one passenger can be indirectly calculated. The heart rate variability data can serve as an objective and continuous indicator for assessing the passenger's physiological stress level and motion sickness pressure level, providing important auxiliary physiological basis for the dynamic judgment of motion sickness risk level.
[0052] Regarding step S102, in one embodiment, obtaining the motion sickness risk level of the vehicle includes: determining the vehicle's motion intensity index based on the vehicle's motion data; and determining the motion sickness risk level based on the motion intensity index, the status data, and the in-vehicle environment data.
[0053] Specifically, based on the vehicle's longitudinal acceleration, lateral acceleration, and yaw rate, a motion intensity index is determined. This index quantifies the intensity or complexity of the vehicle's current motion. Then, by combining the determined motion intensity index with data reflecting passenger physiological and behavioral responses (such as visual attention and / or heart rate variability) and in-vehicle environmental data reflecting air quality (such as CO2 concentration), a comprehensive assessment of motion sickness risk level is achieved. This surpasses assessments based on a single motion dimension, enabling a more comprehensive and accurate analysis of motion sickness risk.
[0054] In one embodiment, the vehicle motion data includes the vehicle's longitudinal acceleration, lateral acceleration, and yaw rate; determining the vehicle's motion intensity index based on the vehicle motion data includes: determining the vehicle's motion intensity index using the following formula based on the vehicle's longitudinal acceleration, lateral acceleration, and yaw rate:
[0055]
[0056] In the formula, This is the exercise intensity index. , , To preset normalized weights, For longitudinal acceleration, For lateral acceleration, ω represents the yaw rate.
[0057] Specifically, the longitudinal acceleration, lateral acceleration, and yaw rate of the vehicle are obtained and substituted into the above formula. This formula integrates the effects of longitudinal acceleration, lateral acceleration, and yaw rate, and can effectively characterize the overall intensity of vehicle motion and determine the motion intensity index.
[0058] In one embodiment, determining the motion sickness risk level based on the exercise intensity index, the status data, and the in-vehicle environment data includes: inputting the exercise intensity index, the status data, and the in-vehicle environment data into a pre-trained risk assessment model to output a real-time motion sickness risk index; and determining the motion sickness risk level based on the real-time motion sickness risk index and a preset threshold range.
[0059] Specifically, the motion intensity index, state data, and in-vehicle environment data are used as a set of multi-dimensional feature vectors and input into a pre-trained risk assessment model. The risk assessment model can use gradient boosting decision trees or support vector machines, and outputs a quantitative real-time motion sickness risk index (RRSI).
[0060] In a specific example, the Real-Time Motion Sickness Risk Index (RRSI) is a value ranging from 0 to 100, with higher values indicating a greater risk of motion sickness. The risk level is determined by comparing the RRSI to a preset threshold range. The preset threshold ranges are shown in Table 1.
[0061] Table 1
[0062] Real-time motion sickness risk index Motion sickness risk level 0≤RRSI<30 Low risk 30≤RRSI<60 Medium risk 60≤RRSI<100 High risk
[0063] Regarding step S103, in one embodiment, controlling the vehicle to execute a corresponding anti-motion sickness control strategy based on the motion sickness risk level includes: controlling at least one of the vehicle's air conditioning unit, fresh air unit, and in-vehicle fragrance generator based on the motion sickness risk level, so as to control the vehicle to execute the corresponding anti-motion sickness control strategy.
[0064] Specifically, the vehicle's air conditioning unit, fresh air system, and in-vehicle fragrance generator together constitute a multi-sensory collaborative intervention execution system. The air conditioning unit includes air conditioning vents, which use vector blades driven by stepper motors. These blades can be precisely positioned in both horizontal and vertical directions and can independently control the wind speed of each vent, thus achieving fine control over the airflow direction.
[0065] The fresh air system includes a controllable fresh air intake valve and an in-vehicle air circulation valve, which can be adjusted to manage the cleanliness of the air inside the vehicle.
[0066] The car fragrance generator contains a variety of selected natural fragrance capsules with soothing or invigorating effects (such as peppermint, ginger, and lemongrass). Through a micro-pump and independent air duct, the selected scent molecules can be precisely and controllably released into the passenger compartment.
[0067] Based on the risk level of motion sickness, one or more of the air conditioning unit, fresh air unit, and in-vehicle fragrance generator are controlled to achieve comprehensive and proactive adjustment of airflow, temperature, air quality and olfactory environment. Before passengers feel obvious discomfort, the vehicle is proactively controlled to implement corresponding anti-motion sickness control strategies, thereby effectively improving riding comfort and enhancing the riding experience.
[0068] In one embodiment, the motion sickness risk level includes low risk, medium risk, or high risk; controlling the vehicle to execute the corresponding anti-motion sickness control strategy includes: when the motion sickness risk level is low risk, controlling the vehicle to execute a basic comfort strategy, the basic comfort strategy including: controlling the air conditioning vents to be in a preset default position, and / or controlling the air conditioning to execute automatic air sweeping based on a preset sweeping mode; or, when the motion sickness risk level is medium risk, controlling the vehicle to execute a dynamic airflow cues strategy, the dynamic airflow cues strategy including: real-time control of the air conditioning vents' airflow direction to form a preset clamp with the direction of the inertial force generated by the vehicle's current movement. The system may adjust the air conditioning settings, and / or switch the air conditioning's airflow mode to a preset random turbulence mode, which simulates the unpredictable breezes found in nature; or, if the motion sickness risk level is high, control the vehicle to implement a strong intervention and sensory reset strategy, which includes at least one of the following: controlling the air conditioning vents to direct the airflow towards the face or chest area of the at least one passenger, and adjusting the air conditioning temperature to a preset temperature and adjusting the air conditioning fan speed to a preset fan speed; controlling the in-vehicle fragrance generator to release a preset type of refreshing fragrance and maintaining it for a first preset time; and controlling the vehicle's fresh air system to operate at maximum airflow for a second preset time.
[0069] Specifically, when the risk level of motion sickness is low, the vehicle is controlled to implement a basic comfort strategy. The basic comfort strategy is used to maintain a basic level of comfort in the vehicle environment. This mainly includes: placing the air conditioning vents in a preset default position, and / or controlling the air conditioning to operate in a preset sweeping mode. The preset sweeping mode is such as slow, wide-range automatic sweeping to maintain a gentle and uniform air circulation and avoid airflow stagnation.
[0070] When the risk level of motion sickness is medium, the vehicle is controlled to implement a dynamic airflow cueing strategy. This strategy utilizes dynamic airflow to provide passengers with mechanical cues that are coordinated with the vehicle's movement, thereby alleviating sensory conflict between the vestibular and visual systems. Specifically, this includes:
[0071] Directional cues for airflow: The direction of the inertial force generated by vehicle movement (such as the direction of centrifugal force during turning) is acquired in real time, and the direction of airflow from the air conditioning vents is controlled to form a preset angle with the direction of the inertial force. This preset angle includes angles in opposite directions. For example, when the vehicle is turning left, the airflow is directed to blow from the passenger's right side, thus creating a perceptual cue that aligns with the vehicle's movement.
[0072] Wind field activation: Switch the air conditioning air supply mode to random turbulence mode. Random turbulence mode effectively breaks the stillness and monotony of the air in the car by simulating the ever-changing and directionless breeze in nature, so as to distract passengers' sensory attention and reduce the focus on motion discomfort.
[0073] When the risk level of motion sickness is high, the vehicle will implement a strong intervention and sensory reset strategy. This strategy uses powerful and synergistic interventions involving multiple senses (body and smell) to quickly interrupt the escalation of discomfort and reset the passenger's sensory state. The strong intervention and sensory reset strategy includes a combination of one or more of the following measures:
[0074] Targeted airflow and temperature control intervention: Direct the air conditioning vents toward the face or chest area of at least one passenger, while adjusting the air conditioning temperature and fan speed to preset temperatures and fan speeds. The preset temperature is a moderate to cool temperature, and the preset fan speed is a stable and moderate fan speed, in order to provide a strong stimulating effect.
[0075] Activate olfactory intervention: Control the in-car fragrance generator to immediately release a preset type of refreshing fragrance, such as mint or lemongrass, and continue for a first preset time, which can be 30 to 60 seconds, to relieve motion sickness and quickly refresh the mind through olfactory stimulation.
[0076] Airflow control: Controls the vehicle's fresh air system to run at maximum airflow for a second preset time, which can be 3 to 5 minutes, to quickly reduce the carbon dioxide concentration inside the vehicle and improve air freshness, thus alleviating stuffiness from an environmental perspective.
[0077] In one embodiment, after controlling the vehicle to execute a corresponding anti-motion sickness control strategy based on the motion sickness risk level, the method further includes: continuously acquiring the multimodal perception data, and acquiring an updated motion sickness risk level based on the multimodal perception data; and controlling the vehicle to execute a corresponding anti-motion sickness control strategy according to the updated motion sickness risk level.
[0078] Specifically, after controlling the vehicle to implement the corresponding anti-motion sickness control strategy, multimodal perception data is continuously monitored, and the real-time motion sickness risk index (RRSI) is calculated based on the multimodal perception data. The motion sickness risk level is then re-determined based on the real-time motion sickness risk index (RRSI). To prevent frequent strategy switching due to instantaneous data fluctuations, the vehicle is only controlled to implement the corresponding new anti-motion sickness control strategy when the updated motion sickness risk level is inconsistent with the level on which the current anti-motion sickness control strategy is based, and the updated motion sickness risk level remains stable for a period of time (e.g., 2 minutes). This ensures both the stability of the system response and timely intervention in the riding experience.
[0079] In one embodiment, after controlling the vehicle to execute the corresponding motion sickness control strategy, the method further includes: recording data during the execution of the motion sickness control strategy, wherein the data includes at least multimodal perception data, motion intensity index, real-time motion sickness risk index, motion sickness risk level, and motion sickness control strategy; and optimizing the risk assessment model or the motion sickness control strategy based on the recorded data.
[0080] Specifically, after each time the vehicle executes a corresponding anti-motion sickness control strategy, the system records key data during the execution of that strategy, such as multimodal perception data, motion intensity index, real-time motion sickness risk index, the final determined motion sickness risk level, and the executed anti-motion sickness control strategy and its parameters. Based on this data, the risk assessment model is continuously retrained and its parameters are fine-tuned, or the execution parameters and duration of each level of anti-motion sickness control strategy are iteratively optimized. In this way, as usage time increases, risk assessment can be adaptively made more accurate, and control interventions can be more tailored to the individual reactions and needs of different passengers, thereby achieving long-term personalized comfort improvements.
[0081] It should be noted that although the steps in the above embodiments are described in a specific order, those skilled in the art will understand that in order to achieve the effect of this application, different steps do not necessarily have to be executed in such an order. They can be executed simultaneously (in parallel) or in other orders, and these variations are all within the scope of protection of this application.
[0082] Those skilled in the art will understand that all or part of the processes in the method of the above-described embodiment can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable file, or some intermediate form. The computer-readable storage medium can include any entity or device capable of carrying the computer program code, a medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory, a random access memory, an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.
[0083] Furthermore, this application also provides a vehicle control system based on multimodal perception.
[0084] See appendix Figure 3 , Figure 3 This is a schematic diagram of a multimodal perception-based vehicle control system according to an embodiment of this application; as shown... Figure 3 As shown, the vehicle control system based on multimodal perception includes a three-layer architecture: a multimodal perception module (input layer), a decision layer, and an execution layer. The multimodal perception module, as the input layer, is responsible for real-time acquisition of multimodal perception data, specifically including:
[0085] The vehicle motion sensing unit is used to acquire vehicle motion data, including longitudinal acceleration, lateral acceleration, and yaw rate.
[0086] A visual attention sensor is used to acquire the direction of the passenger's visual attention.
[0087] The environmental sensing unit is used to acquire and monitor data on the in-vehicle environment, including carbon dioxide concentration.
[0088] Heart rate sensor, used to acquire heart rate variability data of passengers;
[0089] All multimodal sensing data are then aggregated and uploaded to the decision-making level.
[0090] After receiving the multimodal sensing data, the decision-making level analyzes and judges it according to the following process:
[0091] First, the Motion Intensity Index (MII) is calculated based on vehicle motion data.
[0092] Next, the exercise intensity index, passenger status data, and in-vehicle environment data are input into the pre-trained risk assessment model;
[0093] The real-time motion sickness risk index (RRSI) output by the risk assessment model is mapped by the system to a specific motion sickness risk level, including low risk, medium risk or high risk, according to a preset threshold range.
[0094] Based on the risk level, the decision-making level generates corresponding motion sickness control strategies and distributes them to the execution level.
[0095] The execution layer includes the vehicle's air conditioning system, fresh air system, and in-vehicle fragrance generator. Based on the motion sickness control strategy issued by the decision-making layer, the execution layer independently or collaboratively controls one or more of the above devices, specifically implementing any one of the basic comfort strategy, dynamic airflow cues strategy, or strong intervention and sensory reset strategy.
[0096] The vehicle control system based on multimodal perception also includes a feedback optimization mechanism. After executing the anti-motion sickness control strategy, it continuously records data throughout the entire process, including at least multimodal perception data, motion intensity index, real-time motion sickness risk index, motion sickness risk level, and anti-motion sickness control strategy. Then, it continuously optimizes the parameters of the risk assessment model or the anti-motion sickness control strategy to achieve system self-adaptation and personalized improvement.
[0097] Furthermore, this application also provides a vehicle. In one embodiment of the vehicle according to this application, the vehicle includes a processor and a memory. The memory can be configured to store a program for executing the multimodal perception-based vehicle control method of the above-described method embodiments. The processor can be configured to execute the program in the memory, which includes, but is not limited to, the program for executing the multimodal perception-based vehicle control method of the above-described method embodiments. For ease of explanation, only the parts related to the embodiments of this application are shown. For specific technical details not disclosed, please refer to the method section of the embodiments of this application. The vehicle can be a vehicle formed by various electronic devices. See Appendix Figure 4 , Figure 4 The image exemplarily illustrates a communication connection between memory 11 and processor 12 via a bus.
[0098] Furthermore, this application also provides a computer-readable storage medium. In one embodiment of the computer-readable storage medium according to this application, the computer-readable storage medium can be configured to store a program that executes the multimodal perception-based vehicle control method of the above-described method embodiments. This program can be loaded and run by a processor to implement the above-described multimodal perception-based vehicle control method. For ease of explanation, only the parts related to the embodiments of this application are shown; for specific technical details not disclosed, please refer to the method section of the embodiments of this application. The computer-readable storage medium can be a memory device formed by various electronic devices. Optionally, in the embodiments of this application, the computer-readable storage medium is a non-transitory computer-readable storage medium.
[0099] Furthermore, it should be understood that since the various modules are only provided to illustrate the functional units of the device described in this application, the physical devices corresponding to these modules may be the processor itself, or a part of the processor's software, hardware, or a combination of both. Therefore, the number of modules shown in the figures is merely illustrative.
[0100] Those skilled in the art will understand that the various modules in the device can be adaptively split or combined. Such splitting or combining of specific modules will not cause the technical solution to deviate from the principles of this application; therefore, the technical solutions after splitting or combining will fall within the protection scope of this application.
[0101] The technical solutions of this application have been described above with reference to the preferred embodiments shown in the accompanying drawings. However, it will be readily understood by those skilled in the art that the scope of protection of this application is obviously not limited to these specific embodiments. Without departing from the principles of this application, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will all fall within the scope of protection of this application.
Claims
1. A vehicle control method based on multimodal perception, characterized in that, The method is applied to a vehicle, the vehicle being equipped with a multimodal perception module; the method includes: Based on the multimodal perception module, multimodal perception data is acquired, wherein the multimodal perception data includes at least vehicle motion data, the status data of at least one passenger, and in-vehicle environment data; Based on the multimodal perception data, the motion sickness risk level of the vehicle is obtained; Based on the motion sickness risk level, the vehicle is controlled to implement the corresponding motion sickness prevention control strategy.
2. The vehicle control method based on multimodal perception according to claim 1, characterized in that, The multimodal perception module includes a vehicle motion perception unit, a visual attention sensor, and an environmental perception unit; The acquisition of multimodal sensing data includes: Based on the vehicle motion sensing unit, vehicle motion data is acquired, including the vehicle's longitudinal acceleration, lateral acceleration, and yaw rate. Based on the visual attention sensor, the state data of at least one passenger is acquired, and the state data includes at least the visual attention direction; Based on the environmental sensing unit, in-vehicle environmental data is acquired, including at least the in-vehicle carbon dioxide concentration.
3. The vehicle control method based on multimodal perception according to claim 2, characterized in that, The multimodal sensing module also includes a heart rate sensor; The acquisition of multimodal sensing data also includes: Based on the heart rate sensor, status data of at least one passenger is acquired, including heart rate variability data of at least one passenger.
4. The vehicle control method based on multimodal perception according to claim 1, characterized in that, The step of controlling the vehicle to execute a corresponding anti-motion sickness control strategy based on the motion sickness risk level includes: Based on the motion sickness risk level, at least one of the vehicle's air conditioning system, fresh air system, and in-vehicle fragrance generator is controlled to control the vehicle to execute the corresponding anti-motion sickness control strategy.
5. The vehicle control method based on multimodal perception according to claim 1, characterized in that, After controlling the vehicle to implement a corresponding anti-motion sickness control strategy based on the motion sickness risk level, the method further includes: The multimodal perception data is continuously acquired, and an updated motion sickness risk level is obtained based on the multimodal perception data. Based on the updated motion sickness risk level, the vehicle is controlled to implement the corresponding motion sickness prevention control strategy.
6. The vehicle control method based on multimodal perception according to claim 1 or 5, characterized in that, Obtain the motion sickness risk level of the vehicle, including: Based on the vehicle motion data, the vehicle's motion intensity index is determined; Based on the exercise intensity index, the status data, and the in-vehicle environment data, the motion sickness risk level is determined.
7. The vehicle control method based on multimodal perception according to claim 6, characterized in that, The vehicle motion data includes the vehicle's longitudinal acceleration, lateral acceleration, and yaw rate; The determination of the vehicle's motion intensity index based on the vehicle motion data includes: Based on the vehicle's longitudinal acceleration, lateral acceleration, and yaw rate, the vehicle's motion intensity index is determined using the following formula: In the formula, This is the exercise intensity index. , , To preset normalized weights, For longitudinal acceleration, For lateral acceleration, For yaw rate; and / or, The determination of motion sickness risk level based on the exercise intensity index, the status data, and the in-vehicle environment data includes: The exercise intensity index, the state data, and the in-vehicle environment data are input into a pre-trained risk assessment model to output a real-time motion sickness risk index. The motion sickness risk level is determined based on the real-time motion sickness risk index and the preset threshold range.
8. The vehicle control method based on multimodal perception according to claim 4 or 5, characterized in that, The motion sickness risk level includes low risk, medium risk, or high risk; the control of the vehicle to execute the corresponding anti-motion sickness control strategy includes: When the motion sickness risk level is low, the vehicle is controlled to execute a basic comfort strategy, which includes: controlling the air conditioning vents to be in a preset default position, and / or controlling the air conditioning to perform automatic air swing based on a preset swing mode; or, When the motion sickness risk level is medium, the vehicle is controlled to execute a dynamic airflow cuing strategy. This strategy includes: real-time control of the air conditioning vents to form a preset angle with the direction of the inertial force generated by the vehicle's current movement; and / or switching the air conditioning's airflow mode to a preset random turbulence mode, which simulates the unpredictable breezes found in nature; or... When the motion sickness risk level is high, the vehicle is controlled to implement a strong intervention and sensory reset strategy, which includes at least one of the following: Control the airflow direction of the air conditioning vents to the face or chest area of at least one passenger, and adjust the air conditioning temperature to the preset temperature and adjust the air conditioning fan speed to the preset fan speed; Control the car's fragrance generator to release a preset type of refreshing fragrance and maintain it for the first preset time; The vehicle's fresh air system is controlled to operate at maximum airflow for a second preset time.
9. The vehicle control method based on multimodal perception according to claim 7, characterized in that, After controlling the vehicle to execute the corresponding anti-motion sickness control strategy, the method further includes: Record data during the execution of the motion sickness control strategy, wherein the data includes at least multimodal perception data, motion intensity index, real-time motion sickness risk index, motion sickness risk level, and motion sickness control strategy; Based on the recorded data, the risk assessment model or the motion sickness control strategy is optimized.
10. A vehicle comprising a processor and a memory, the memory being adapted to store a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by the processor to perform the vehicle control method based on multimodal perception as described in any one of claims 1 to 9.
11. A computer-readable storage medium storing a plurality of program codes, characterized in that, The program code is adapted to be loaded and run by a processor to perform the vehicle control method based on multimodal perception as described in any one of claims 1 to 9.