Immersive interaction method and system based on context awareness and carrier moving state
By constructing scenario feature vectors and dynamically adjusting multimodal interaction modes, the problems of insufficient immersive experience and latency in in-vehicle interaction systems are solved, enabling safe and efficient immersive interaction in dynamic environments, adapting to driving risks and user preferences.
Patent Information
- Application Number
- CN202511975214.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-12-25
- Publication Date
- 2026-03-31
AI Technical Summary
Existing in-vehicle interactive systems lack multimodal channel collaborative scheduling, resulting in insufficient immersive experience, high motion photon latency, and an inability to dynamically adapt to driving risks and environmental changes, thus affecting driving safety.
By collecting data on vehicle movement, environment, and user status, a scenario feature vector is constructed, immersion allowance is calculated, multimodal interaction modes are dynamically adjusted, timing alignment and latency compensation are performed, and interaction latency is optimized by combining multi-source sensors and predictive rendering technology.
It achieves multimodal timing alignment and delay compensation in dynamic environments, reduces motion photon delay, enhances immersive experience, ensures driving safety, and adapts to different driving risks and user preferences.
Smart Images
Figure CN121767600A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of intelligent transportation and human-computer interaction technology, specifically to an immersive interaction method and system based on context awareness and vehicle movement status. Background Technology
[0002] With the maturity of autonomous driving technology and the rapid development of information networks, vehicles are evolving from transportation tools into mobile intelligent spaces. User behavior inside the car is no longer limited to driving operations but extends to multiple scenarios including work, entertainment, social interaction, and life services. However, existing in-vehicle interaction systems are still mainly based on touchscreens and voice assistants, and have the following problems: The problem of a single interaction mode: due to the lack of coordinated scheduling of multimodal channels such as vision, hearing, touch, and body sensation, it is difficult to build a truly immersive experience, and the immersive experience needs improvement; the problem of an imbalance between safety and immersion: in high-risk driving scenarios, highly immersive content may still be pushed, affecting driver attention or causing motion sickness, thus affecting driving safety; the problem of asynchronous timing: there is a delay between AR / VR rendering and the actual movement of the vehicle, resulting in excessive motion-to-photon latency, causing user discomfort; the problem of a lack of dynamic adaptability: it cannot dynamically adjust the interaction strategy according to multiple factors such as real-time road conditions, lighting and noise, and passenger physiological state.
[0003] While some parks have experimented with virtual reality (VR) for fixed-path amusement facilities like roller coasters in theme parks, these solutions rely on highly controlled physical environments and are unsuitable for random driving scenarios on open roads. Furthermore, the rapid development of location-based augmented reality (AR) applications in recent years has demonstrated the immense potential of using the real world as an interactive canvas—through real-world map modeling, spatial positioning, and narrative design—to construct new social entertainment experiences that transcend physical and digital boundaries. However, in the field of intelligent driving, there is currently no systematic solution that can safely and seamlessly transfer these concepts to autonomous vehicles, enabling large-scale outdoor immersive interaction while ensuring driving safety. Therefore, there is an urgent need for a comprehensive in-vehicle immersive interaction system and method that integrates dynamic context perception, risk assessment, multimodal synchronization, and intelligent content scheduling. Summary of the Invention
[0004] The purpose of this invention is to provide an immersive interaction method and system based on context awareness and vehicle movement status, so as to solve the existing technical problems in the background art.
[0005] To address the aforementioned technical problems, the present invention provides the following technical solution: an immersive interaction method based on context awareness and vehicle movement status, comprising the following steps: Collect data on vehicle movement status, environment, and user status; Analyze and process the data, and output a scenario feature vector; Based on the context feature vector, a data model is constructed and the immersion allowance is calculated; Define immersive interaction mode levels, determine the immersive interaction mode level based on immersion tolerance, and dynamically adjust it; Based on the level of immersive interaction mode, select the corresponding multimodal terminal optimization combination; Based on the interaction delays existing in the optimized combination and operation of multimodal terminals, multimodal synchronization compensation with timing alignment and delay compensation is performed.
[0006] Based on the above technical solution, the output of the scenario feature vector includes: Data is collected using multi-source sensors, and the data is synchronized and filtered in time. Output a structured scenario feature vector, where scenario features include driving status, road type, and traffic risk level R. d ∈[0,1], environmental factor E, and user attention state; Among them, driving status includes autonomous driving L3+ and manual driving; road type includes highway, city, scenic area and tunnel; traffic risk level is judged based on distance to the vehicle in front, lateral acceleration and intersection complexity; environmental factors include light intensity and background noise; user attention status is judged by eye tracking to determine whether the user is looking ahead.
[0007] Based on the above technical solution, the calculation of the immersion allowance includes: Building a user model: Based on federated learning, a personalized preference model P is built to record the user's acceptance threshold for cultural explanations, games, tourism, and meditation content types; Construct a scene recognition model: Combine convolutional neural network (CNN) and Transformer model for visual semantic segmentation, and integrate POI map data to identify the current scene category, which includes historical sites, business districts, and natural scenic spots; An immersion rating engine is built to calculate the immersion allowance I.
[0008] Based on the above technical solution, the immersion tolerance I is calculated by the following formula: I = σ ( w 1(1 R d )+ w 2(1 E illuminance )+ w 3 P +w 4(1 A attention )) Where σ is the Sigmoid function, R d Traffic risk level, E illuminance Ambient lighting, P is the user preference score, and A is the user preference score. attention For the state of attention, w i These are learnable weights.
[0009] Based on the above technical solution, the definition of the immersive interaction mode level and the selection of the multimodal terminal optimization combination include: When the immersion allowance I ≥ 0.8, it is determined that the current mode is a fully immersive interaction mode, and the selected terminal combination is VR glasses, haptic chair and surround sound. When the immersion allowance is 0.6≤I<0.8, it is determined that the current mode is semi-immersive interaction mode, and the selected terminal combination is augmented reality head-up display (AR HUD), in-vehicle holographic projection, and zoned audio. When the immersion tolerance is 0.3≤I<0.6, it is determined that the current mode is basic immersive interaction mode, and the selected terminal combination is head-up display (HUD) and voice prompts. When the immersion allowance I < 0.3 or manual driving mode is selected, the current mode is determined to be safe, and the selected terminal combination to be enabled is navigation voice and collision warning.
[0010] Based on the above technical solution, the dynamic adjustment of the immersive interaction mode level includes: The immersive interaction mode level is adjusted based on the hysteresis mechanism. After the two adjacent immersive interaction modes are switched, the immersive interaction mode level needs to be adjusted again after the immersion allowance is below or above the threshold and remains at that level for a predetermined time.
[0011] Based on the above technical solution, the multimodal terminal optimizes the multimodal synchronization compensation during operation, supporting motion photon delay ≤50ms, including the following steps: Predictive rendering: Based on the fusion technology of inertial measurement unit (IMU) and global navigation satellite system (GNSS), the vehicle state is predicted 0.1 seconds in advance, and the corresponding viewpoint is rendered in advance; Frame interpolation: Inserting intermediate frames when network packet loss or high GPU load occurs; Audio-visual synchronization correction: Utilizes a microphone array to detect lip movement timestamps and dynamically adjusts audio playback offset.
[0012] Based on the above technical solution, the multimodal synchronization compensation during the optimized operation of the multimodal terminal further includes: Configure a Quality of Service (QoS) feedback loop to collect vehicle speed and network status in real time, and dynamically adjust the Field of View (FOV) and rendering resolution parameter Resolution according to the following dynamic formula: FOV = max(60°, Base_FOV (1-Speed / 100)), where Base_FOV is the system's preset base field of view, Speed is the vehicle's real-time speed, obtained through GNSS or a vehicle speed sensor, 60° is the lower limit threshold for the safe field of view, and 100 is the speed normalization coefficient, mapping the vehicle's real-time speed to the range [0,1], i.e., Speed / 100∈[0,1]; Resolution = Base_Res (1-QoS score / 2), where Base_Res is the system's preset base resolution, QoS score The network service quality score is calculated, with 2 representing a dynamic scaling factor. The obtained field of view and rendering resolution are smoothed by first-order low-pass filtering and then output.
[0013] Based on the above technical solution, in the immersive interaction mode, it receives instructions to trigger security risk events and executes security control and compliance operations.
[0014] Based on the above technical solution, the security control and compliance operation includes: Forced Degradation: When the CAN bus detects braking, steering signals, or the driver's hands on the steering wheel, the system immediately triggers the safety mode and records the reason for the degradation and the timestamp. In safety mode, the system only retains navigation voice and collision warning functions to ensure that critical information is not obscured. After manual driving intervention, the system will continuously monitor the driving status. When a stable driving status is detected for 10 consecutive seconds, the system will gradually restore the immersion mode according to the immersion allowance I value. Emergency Interruption: Set up an emergency interruption interface to support the V2X vehicle network to interrupt all immersive interactive content and pop up a warning message when it receives a warning of an accident ahead; Privacy protection: Privacy data is stored locally in encrypted form and uses differential privacy upload statistics.
[0015] This application also provides an immersive interaction system based on context awareness and vehicle movement status, including: The context awareness module is used to collect data on the vehicle, environment, and user status, analyze and process the data, and output context feature vectors. The data analysis module is used to build a data model and calculate the immersion allowance based on the context feature vector; The immersion adjustment module is used to define the level of immersion interaction mode, determine the level of immersion interaction mode based on the immersion allowance, and make dynamic adjustments. The multimodal synchronization module is used to select the optimal combination of multimodal terminals according to the level of immersive interaction mode; and to perform multimodal synchronization compensation with timing alignment and delay compensation based on the interaction delay that exists during the operation of the multimodal terminal optimization combination. The security control module is used to receive security event triggering instructions and perform security control and compliance operations in immersive interactive mode.
[0016] Based on the above technical solutions, a data access module is also included, which supports third-party content providers in creating security metadata tags for maximum field of view occupancy, interruption flags, and interaction duration, for automatic system scheduling and risk assessment.
[0017] This application also provides a computer system, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the immersive interaction method based on context awareness and vehicle movement state.
[0018] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the immersive interaction method based on context awareness and vehicle movement state.
[0019] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the immersive interaction method based on context awareness and vehicle movement state.
[0020] The beneficial effects of the technical solution provided by this invention are as follows: This invention provides an immersive interaction method based on context awareness and vehicle movement status. It enables adaptive adjustment of the context-driven immersive interaction mode, dynamically calculating the immersion allowance based on driving traffic risks, environmental conditions, user preferences, and physiological state, and automatically determining and switching the immersive interaction mode level accordingly. It solves the problems of multimodal temporal alignment and latency compensation under dynamic in-vehicle motion by effectively reducing motion-to-photon latency and alleviating motion sickness through predictive rendering, frame interpolation compensation, and latency optimization. Simultaneously, it provides effective protection for driving safety by downgrading the immersive interaction mode upon detecting a safety risk event or human intervention, ensuring that critical information is not obscured, making it safer and more efficient. Attached Figure Description
[0021] Figure 1 This is a flowchart illustrating the present invention; Detailed Implementation The present invention will be further described below with reference to embodiments: like Figure 1 As shown, this application provides an immersive interaction method based on context awareness and vehicle movement state, including the following steps: Collect data on vehicle movement status, environment, and user status; Analyze and process the data, and output a scenario feature vector; Based on the context feature vector, a data model is constructed and the immersion allowance is calculated; Define immersive interaction mode levels, determine the immersive interaction mode level based on immersion tolerance, and dynamically adjust it; Based on the level of immersive interaction mode, select the corresponding multimodal terminal optimization combination; Based on the interaction delays existing in the optimized combination and operation of multimodal terminals, multimodal synchronization compensation with timing alignment and delay compensation is performed.
[0022] This invention provides an immersive interaction method based on context awareness and vehicle movement status. It enables adaptive adjustment of the context-driven immersive interaction mode, dynamically calculating the immersion allowance based on driving traffic risks, environmental conditions, user preferences, and physiological state, and automatically determining and switching the immersive interaction mode level accordingly. It solves the problems of multimodal temporal alignment and latency compensation under dynamic in-vehicle motion by effectively reducing motion-to-photon latency and alleviating motion sickness through predictive rendering, frame interpolation compensation, and latency optimization. Simultaneously, it provides effective protection for driving safety by downgrading the immersive interaction mode upon detecting a safety risk event or human intervention, ensuring that critical information is not obscured, making it safer and more efficient.
[0023] Furthermore, this application constructs an open content ecosystem, supporting third-party content providers in creating standardized scenario packages containing security metadata and breakpoint information, facilitating system scheduling and enhancing adaptability. It also improves the inclusive experience for special needs groups by providing visually and hearing impaired users with multimodal navigation and entertainment services based on haptic feedback, temperature control, and spatial audio. Specifically, it integrates personalized preference models based on the accessibility needs of special needs groups, outputting corresponding user preference scores and dynamically adjusting weights. w 3. Select the accessibility mode based on the immersion allowance calculation results.
[0024] Based on the above technical solution, the output of the scenario feature vector includes: Data is collected using multi-source sensors, and the data is synchronized and filtered in time. Output a structured scenario feature vector, where scenario features include driving status, road type, and traffic risk level R. d∈[0,1], environmental factor E, and user attention state; Among them, driving status includes autonomous driving L3+ and manual driving; road type includes highway, city, scenic area and tunnel; traffic risk level is judged based on distance to the vehicle in front, lateral acceleration and intersection complexity; environmental factors include light intensity and background noise; user attention status is judged by eye tracking to determine whether the user is looking ahead.
[0025] Based on the above technical solution, the calculation of the immersion allowance includes: Building a user model: Based on federated learning, a personalized preference model is built to record the user's acceptance threshold for news, entertainment, tour guide, mental and physical adjustment, social interaction, and educational content types, and output the user preference score P. Specifically, the content types may include cultural explanations, games, tourism, meditation, and other content.
[0026] Construct a scene recognition model: Combine convolutional neural network (CNN) and Transformer model for visual semantic segmentation, and integrate POI map data to identify the current scene category, which includes historical sites, business districts, and natural scenic spots; An immersion scoring engine is constructed, which calculates the immersion allowance I from the structured context feature vector and the user preference score.
[0027] Based on the above technical solution, the immersion tolerance I is calculated by the following formula: I = σ ( w 1(1 R d )+ w 2(1 E illuminance )+ w 3 P + w 4(1 A attention )) Where σ is the Sigmoid function, R d Traffic risk level, E illuminance Ambient lighting, P is the user preference score, and A is the user preference score. attention For the state of attention, w i These are learnable weights.
[0028] This application is the first to propose classifying traffic risk level R. dUser attention state (A), ambient light (E), and historical user preference score (P) are jointly modeled as a continuous variable, immersion allowance (I), and serve as the core key factor in determining the level of immersive interaction mode.
[0029] Based on the above technical solution, the definition of the immersive interaction mode level and the selection of the multimodal terminal optimization combination include: When the immersion allowance I ≥ 0.8, it is determined that the current mode is a fully immersive interaction mode, and the selected terminal combination is VR glasses, haptic chair and surround sound. When the immersion allowance is 0.6≤I<0.8, it is determined that the current mode is semi-immersive interaction mode, and the selected terminal combination is augmented reality head-up display (AR HUD), in-vehicle holographic projection, and zoned audio. When the immersion tolerance is 0.3≤I<0.6, it is determined that the current mode is basic immersive interaction mode, and the selected terminal combination is head-up display (HUD) and voice prompts. When the immersion allowance I < 0.3 or manual driving mode is selected, the current mode is determined to be safe, and the selected terminal combination to be enabled is navigation voice and collision warning.
[0030] Preferably, this application adopts a dual control mechanism of immersion rating system, namely immersion allowance and safety degradation system, which realizes quantitative management and risk avoidance of in-vehicle immersive interactive experience, resulting in a better user experience and effective guarantee of safety performance.
[0031] Based on the above technical solution, the dynamic adjustment of the immersive interaction mode level includes: The immersive interaction mode level is adjusted based on the hysteresis mechanism. After the two adjacent immersive interaction modes are switched, the immersive interaction mode level needs to be adjusted again after the immersion allowance is below or above the threshold and remains at that level for a predetermined time.
[0032] By introducing a hysteresis mechanism during the dynamic adjustment of the immersive interaction level, frequent mode switching caused by road condition fluctuations is avoided, and the switching frequency is limited to prevent oscillations in the immersive interaction mode, thereby improving the stability of the user experience and increasing user satisfaction by 23.7%. At the same time, it reduces the wear and tear of frequent device startups, which is beneficial to protecting the device's lifespan. For example, when the immersive interaction mode is adjusted from the fully immersive interaction mode to the semi-immersive interaction mode, the immersion allowance must be lower than the threshold of 0.75 and remain there for a predetermined time of 5 seconds before the upgrade can be allowed again.
[0033] Based on the above technical solution, the multimodal terminal optimizes the multimodal synchronization compensation during operation, supporting motion photon delay ≤50ms, including the following steps: Predictive rendering: Based on the fusion technology of inertial measurement unit (IMU) and global navigation satellite system (GNSS), the vehicle state is predicted 0.1 seconds in advance, and the corresponding viewpoint is rendered in advance; Frame interpolation: Inserting intermediate frames when network packet loss or high GPU load occurs; Audio-visual synchronization correction: Utilizes a camera to detect lip movements, a microphone array to correct audio, and dynamically adjusts audio playback offset.
[0034] By using a camera and microphone to dynamically calculate and compensate for offsets in real time, a truly immersive experience can be achieved compared to traditional simple fixed offset compensation.
[0035] Based on the above technical solution, the multimodal synchronization compensation during the optimized operation of the multimodal terminal further includes: Configure a Quality of Service (QoS) feedback loop to collect vehicle speed and network status in real time, and dynamically adjust the Field of View (FOV) and rendering resolution parameter Resolution according to the following dynamic formula: FOV = max(60°, Base_FOV (1-Speed / 100)), where Base_FOV is the system's preset base field of view, Speed is the vehicle's real-time speed, obtained through GNSS or a vehicle speed sensor, 60° is the lower limit threshold for the safe field of view, and 100 is the speed normalization coefficient, mapping the vehicle's real-time speed to the range [0,1], i.e., Speed / 100∈[0,1]; Resolution = Base_Res (1-QoS score / 2), where Base_Res is the system's preset base resolution, QoS score The network service quality score is calculated, with 2 representing a dynamic scaling factor. The obtained field of view and rendering resolution are smoothed by first-order low-pass filtering and then output.
[0036] Actual testing showed that the incidence of motion sickness decreased by 61.5% in high-speed scenarios.
[0037] By employing multimodal synchronization compensation technology—based on multi-sensor data acquisition and predictive rendering, frame interpolation, and audio-visual synchronization correction—multimodal synchronization compensation can be improved without adding new hardware, solving the problem of audio-visual and motion perception misalignment in dynamic in-vehicle environments. Through predictive rendering, frame interpolation, and audio-visual synchronization correction, motion-to-photon latency can be effectively compensated, reducing motion sickness. Test results show that the motion-to-photon latency in this application is controlled within 42ms, significantly better than the industry average of 70ms. A Quality of Service (QoS) feedback loop is also included, adjusting the field of view (FOV) and rendering resolution according to vehicle speed, effectively reducing motion sickness and lowering the probability of motion sickness. Furthermore, through the QoS feedback loop and dynamic adjustment of the immersive interaction mode level, a basic user experience is maintained even under bandwidth fluctuations. Based on the above technical solution, in the immersive interaction mode, it receives instructions to trigger security risk events and executes security control and compliance operations.
[0038] Based on the above technical solution, the security control and compliance operation includes: Forced Degradation: When the CAN bus detects braking, steering signals, or the driver's hands on the steering wheel, the system immediately triggers the safety mode and records the reason for the degradation and the timestamp. In safety mode, the system only retains navigation voice and collision warning functions to ensure that critical information is not obscured. After manual driving intervention, the system will continuously monitor the driving status. When a stable driving status is detected for 10 consecutive seconds, the system will gradually restore the immersion mode according to the immersion allowance I value. Emergency Interruption: Set up an emergency interruption interface to support the V2X vehicle network to interrupt all immersive interactive content and pop up a warning message when it receives a warning of an accident ahead; Privacy protection: Privacy data is stored locally in encrypted form and uses differential privacy upload statistics.
[0039] It should be noted that the use of differential privacy upload statistical features complies with GDPR and ISO / SAE 21434 standards. The three security control and compliance operation steps described above are triggered based on different current conditions, such as driver operation, external time, and data processing needs, and can be executed independently, offering greater adaptability.
[0040] Preferably, this application can also effectively ensure driving safety. When a high-risk event is detected or when manual driving is required, a safety mode is activated, that is, the system is forcibly downgraded to HUD or voice prompts and collision warnings to ensure that critical information is not obscured.
[0041] This application also provides an immersive interaction system based on context awareness and vehicle movement status, including: The context awareness module is used to collect data on the vehicle, environment, and user status, analyze and process the data, and output context feature vectors. The data analysis module is used to build a data model and calculate the immersion allowance based on the context feature vector; The immersion adjustment module is used to define the level of immersion interaction mode, determine the level of immersion interaction mode based on the immersion allowance, and make dynamic adjustments. The multimodal synchronization module, including a predictive rendering unit and a frame interpolation unit, is used to select the optimal combination of multimodal terminals according to the level of immersive interaction mode; and to perform multimodal synchronization compensation with timing alignment and delay compensation based on the interaction delay that exists during the operation of the multimodal terminal optimization combination. The security control module is used to receive security event triggering instructions and perform security control and compliance operations in immersive interactive mode.
[0042] Based on the above technical solutions, a data access module is also included, which supports third-party content providers in creating security metadata tags for maximum field of view occupancy, interruption flags, and interaction duration, for automatic system scheduling and risk assessment.
[0043] Preferably, by incorporating a data access module, it can support standardized access to third-party content, resulting in better adaptability, a more diversified user experience, and promising practical prospects.
[0044] More preferably, the multimodal immersive interactive system based on context awareness and vehicle movement status provided in this application is compatible with the current aftermarket and OEM markets. It can be integrated into traditional vehicles as an independent domain controller or deployed as a native function of OEMs.
[0045] The multimodal immersive interaction system and method of this application are illustrated by the following embodiments: Example 1: Augmented Reality Tours in Cultural Tourism When a vehicle enters the electronic fence area of Huangshan Scenic Area in autonomous driving mode; The context-aware module identified it as a low-speed, open road. d =0.2; The data analysis module retrieves the Huangshan AR scene package, which includes ancient building restoration models and audio guide scripts; User preferences indicate a preference for historical and cultural content, P=0.9; The calculated value of I is 0.82, triggering the fully immersive interaction mode; AR glasses overlay a virtual reconstruction of the Huangshan cloud sea and the Welcoming Pine, the seats vibrate slightly with the rhythm of the mountain wind, and Huangmei Opera background music plays in sync. If a pedestrian suddenly crosses the road ahead, the system will switch to a red-framed HUD warning and voice announcement within 0.3 seconds, pausing the immersive content.
[0046] Example 2: Gamified Commuting Experience During the morning rush hour commute, the user selects the City Adventure mode: The system identifies the current location as a congested urban road section. d =0.6, I=0.5; Activate the semi-immersive interactive mode to display the virtual mission path on the AR HUD: "Collect 3 energy points along the way"; As the vehicle passes through each intersection, the HUD displays a particle convergence animation and points rewards; If traffic resumes normal, R d When I drops to 0.3 and rises to 0.7, the system automatically unlocks the interactive mini-game "Express Delivery," mapping the vehicle speed to the game character's forward speed. All task rewards can be redeemed for real coffee coupons, with payment authorization completed via the vehicle's hardware security module (HSM).
[0047] More preferably, based on the scene awareness module, data analysis module, immersion adjustment module, multimodal synchronization module, security control module, and data access module in the technical solution of this application, the interactive experience in the following extended scenarios can also be realized: Example 3: Location-Triggered Social Interaction Experience Users participate in the "City Connector" multiplayer AR mission: Based on a world-wide model and real-world map modeling, distributed virtual item delivery tasks are generated. Users wearing AR glasses can see virtual goods and recipient icons in real streets; Each completed delivery unlocks an exclusive digital collectible that can be displayed on social media platforms; The system dynamically adjusts the task difficulty based on weather and crowd density, reducing movement requirements on rainy days; When the vehicle enters a tunnel or a weak signal area, it automatically switches to offline voice narration mode to maintain immersive continuity.
[0048] Example 3 combines real-world geographic data with a narrative-driven interactive mechanism to expand the application scenarios of AR in mobile vehicles.
[0049] Example 4: Barrier-free Multimodal Navigation Visually impaired passengers ride in self-driving taxis: The system identifies its identity and loads the accessibility configuration file; The external landscape is transformed into a 3D spatial audio stream. Hearing birdsong in the left ear indicates that there is a park on the left, and the ticking sound in front indicates the countdown of the traffic light. A slight warming of the left side of the seat indicates an imminent left turn, while vibration on the right side indicates preparation for a right turn. 100 meters before arrival, the tactile feedback frequency increases, and a voice announcement reads: "You have arrived at the South Gate of Shanghai Library." In Example 4, accessibility needs are identified through user modeling, integrated into the construction of a personalized preference model, and the corresponding user preference scores are output with dynamically adjusted weights. w 3. Select the accessibility mode based on the immersion allowance calculation results.
[0050] This application also provides a computer system, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the immersive interaction method based on context awareness and vehicle movement state.
[0051] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the immersive interaction method based on context awareness and vehicle movement state.
[0052] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the immersive interaction method based on context awareness and vehicle movement state.
[0053] The foregoing has shown and described the basic principles and main features of the present invention. It will be apparent to those skilled in the art that the present invention is not limited to the details of the above exemplary embodiments. Therefore, the embodiments should be considered as exemplary and not restrictive. The scope of the present invention is defined by the appended claims rather than the foregoing description. Therefore, it is intended that all variations falling within the meaning and scope of the equivalents of the claims be included within the present invention.
[0054] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.
Claims
1. A method for context-aware and vehicle movement state-based immersive interaction, characterized in that, The method comprises the following steps: Collecting data information of the moving state of the vehicle, the environment, and the user state; Analyzing and processing the data information and outputting a scenario feature vector; Based on the scenario feature vector, a data model is constructed and an immersion allowance is calculated; Defining the level of immersive interaction mode, judging the level of immersive interaction mode according to the immersion allowance and dynamically adjusting it; According to the level of immersive interaction mode, the corresponding selection of the optimized combination of multi-modal terminals is carried out; Based on the interaction delay existing in the running process of the optimized combination of multi-modal terminals, the multi-modal synchronization compensation of time alignment and delay compensation is carried out. 2.The method of claim 1, wherein, The output of the scenario feature vector includes: Data information is collected by using multi-source sensors, and time synchronization and filtering processing are carried out on the data information; outputting a structured situational feature vector, wherein the situational features comprise driving state, road type, traffic risk level R d ∈ [0, 1], environmental factor E, and user attention state A; Among them, the driving state includes automatic driving L3+ and manual driving; the road type includes highway, city, scenic area and tunnel; the traffic risk level is judged based on the distance of the preceding vehicle, lateral acceleration and intersection complexity; the environmental factors include light intensity and background noise; the user attention state is judged by eye tracking whether to gaze at the front. 3.The method of claim 1, wherein, The calculation of the immersion allowance includes: Building a user model: based on federated learning, an individual preference model is built to record the user's acceptance threshold for news information, entertainment, tour guide, physical and mental regulation, social interaction and education content types, and output the user preference score P; Building a scene recognition model: combining convolutional neural network (CNN) and Transformer model for visual semantic segmentation, and fusing POI map data to identify the current scene category, including historical relics, business district and natural scenic area; Building an immersion score engine to calculate the immersion allowance I from the structured scenario feature vector and user preference score.
4. The context-aware and vehicle movement state-based immersive interaction method of claim 1 or 3, wherein, The immersion allowance I is calculated by the following formula: I = σ ( w 1(1 R d )+ w 2(1 E illuminance )+ w 3 P + w 4(1 A attention )) where σ is a Sigmoid function, R d is the traffic risk level, E illuminance is the ambient light, P is the user preference score, A attention is the attention state, w i is a learnable weight. 5.The method of claim 1, wherein, The definition of the level of immersive interaction mode and the selection of the optimized combination of multi-modal terminals include: When the immersion allowance I is greater than or equal to 0.8, it is judged that the current is full immersive interaction mode, and the enabled terminal combination is VR glasses, haptic seat and surround sound effect; When the immersion allowance 0.6≤I<0.8, it is judged that the current is semi-immersive interaction mode, and the enabled terminal combination is augmented reality head-up display (AR HUD), vehicle-mounted holographic projection and zoned audio; When the immersion allowance 0.3≤I<0.6, it is judged that the current is basic immersive interaction mode, and the enabled terminal combination is head-up display (HUD) and voice prompt; When the immersion allowance I is less than 0.3 or in manual driving mode, it is judged that the current is safety mode, and the enabled terminal combination is navigation voice and collision warning.
6. The context-aware and vehicle movement state-based immersive interaction method of claim 1, wherein, The dynamic adjustment of the level of immersive interaction mode includes: Based on the hysteresis mechanism, the level of immersive interaction mode is adjusted, and after the conversion of two adjacent levels of immersive interaction mode, the adjustment of the level of immersive interaction mode continues after the immersion allowance is lower or higher than the threshold value and remains for a predetermined time.
7. The method of claim 1, wherein, The multi-modal synchronization compensation in the running process of the optimized combination of multi-modal terminals supports motion photon delay of less than or equal to 50 ms, which includes the following steps: Predictive rendering: based on the fusion technology of inertial measurement unit (IMU) and global navigation satellite system (GNSS), predict the vehicle state in the next 0.1 seconds, and render the corresponding view in advance; Frame interpolation: insert intermediate frames when network packet loss occurs or the graphics processing unit (GPU) load is high; Audio-visual synchronization correction: use the camera to detect lip movement, and use the microphone array to correct the audio, and dynamically adjust the audio playback offset.
8. The method of claim 7, wherein, The multi-modal terminal optimizes the multi-modal synchronization compensation in the operation process, further comprising: Setting a quality of service (QoS) feedback loop, real-time collection of vehicle speed and network state, and dynamic adjustment of the field of view (FOV) and rendering resolution parameters according to the following dynamic formula: FOV = max(60°, Base_FOV (1 - Speed / 100)), wherein Base_FOV is a system preset base field of view, Speed is a real-time speed of the vehicle, which is obtained through a GNSS or a vehicle speed sensor, 60° is a lower threshold of a safe field of view, and 100 is a speed normalization coefficient, which maps the real-time speed of the vehicle to a range of [0, 1], that is, Speed / 100 ∈ [0, 1]; Resolution = Base_Res (1 - QoS / 2), wherein Base_Res is a system preset base resolution, QoS is a network quality of service score, and 2 is a dynamic scaling coefficient; score score FOV = max(60°, Base_FOV (1 - Speed / 100)), wherein Base_FOV is a system preset base field of view, Speed is a real-time speed of the vehicle, which is obtained through a GNSS or a vehicle speed sensor, 60° is a lower threshold of a safe field of view, and 100 is a speed normalization coefficient, which maps the real-time speed of the vehicle to a range of [0, 1], that is, Speed / 100 ∈ [0, 1]; Resolution = Base_Res (1 - QoS / 2), wherein Base_Res is a system preset base resolution, QoS is a network quality of service score, and 2 is a dynamic scaling coefficient; Smooth the field of view and rendering resolution obtained above by first-order low-pass filtering and output. 9.The method of claim 1, wherein, In the immersive interaction mode, a safety risk event trigger instruction is received, and safety control and compliance operations are performed.
10. The method of claim 9, wherein, The safety control and compliance operations include: Forced degradation: When the CAN bus detects brake, steering signal or driver hand steering, the system immediately triggers a safety mode, records the degradation reason and timestamp, and only retains navigation voice and collision warning functions in the safety mode to ensure that critical information is not obscured. After manual driving intervention, the system will continuously monitor the driving state, and when a stable driving state is detected for 10 seconds, the immersive mode will be gradually restored according to the immersive allowance I value; Emergency interruption: Set an emergency interruption interface to support vehicle-to-everything (V2X) to receive a front accident warning and interrupt all immersive interaction content and pop up warning information; Privacy protection: private data is stored locally and encrypted, and differential privacy is used to upload statistical features.
11. An immersive interaction system based on situational awareness and vehicle movement state, characterized in that, Comprise: A scene awareness module for collecting data information of the vehicle, environment and user state, analyzing and processing the data information, and outputting a scene feature vector; A data analysis module for constructing a data model based on the scene feature vector and calculating an immersive allowance; An immersive adjustment module for defining an immersive interaction mode level, judging the immersive interaction mode level based on the immersive allowance and dynamically adjusting it; A multi-modal synchronization module for selecting a multi-modal terminal optimization combination according to the immersive interaction mode level; And based on the interaction delay existing in the multi-modal terminal optimization combination operation process, a time sequence alignment and delay compensation multi-modal synchronization compensation is performed; A safety control module for receiving a safety event trigger instruction in the immersive interaction mode and performing safety control and compliance operations.
12. The context-aware and vehicle movement state-based immersive interaction system of claim 11, wherein, Further comprising a data access module that supports third-party content providers to create safety metadata tags for maximum field of view occupancy, interruption flags and interaction duration for automatic scheduling and risk assessment by the system.
13. A computer system comprising a memory, a processor and a computer program stored on the memory, wherein the computer program is configured to cause the processor to perform the method of any one of claims 1 to 12. The processor executes the computer program to implement the steps of the immersive interaction method based on context awareness and vehicle movement state according to any one of claims 1 to 10.
14. A computer readable storage medium having stored thereon a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the immersive interaction method based on context awareness and vehicle movement state according to any one of claims 1 to 10.
15. A computer program product comprising a computer program, characterized in that, The computer program, which is executed by a processor, implements the steps of the context-aware and vehicle movement state-based immersive interaction method according to any one of claims 1 to 10.