Scene-awareness-oriented acoustic control and multi-mode prompting method in intelligent cabin and related system

Abstract

The invention discloses a scene perception-oriented acoustic control and multi-mode prompting method in an intelligent cabin and a related system. The method comprises the following steps: monitoring and identifying a predefined trigger event in real time, collecting vehicle dynamic data, external environment data and driver state data, and carrying out multi-source data fusion; based on a fused result, identifying and outputting a driving scene of the current vehicle through a large model, and calling an acoustic management strategy and a multi-mode prompting strategy matched with the driving scene from a preset strategy library based on the driving scene; when a concurrent trigger event occurs, according to the risk level of the driving scene and the priority of the concurrent trigger event, dynamically adjusting the priority and prompt resource allocation of a plurality of concurrent prompt operations in the multi-mode prompt strategy; and sound field reconstruction, audio source volume adjustment and priority control inside and outside the intelligent cabin are controlled based on an acoustic management strategy, and azimuth output of multi-mode collaborative prompt information is controlled based on an adjusted control strategy. According to the application, the warning reliability and the user acceptability are improved, and the optimal balance between the system energy efficiency and the safety performance is achieved.

Description

Technical Field

[0001] This application relates to the fields of automotive intelligent cockpit and driving safety technology, specifically to the technical fields of event triggering, multi-source information fusion and multimodal collaboration, and particularly to a method and related system for acoustic control and multimodal prompting in an intelligent cockpit oriented towards scene perception. Background Technology

[0002] With the rapid development of automotive intelligence and connectivity, the acoustic competition between in-cabin infotainment systems and external environmental warning sounds (such as sirens of special vehicles and vehicle horns) is becoming increasingly fierce. At the same time, drivers face interference from multiple sources of information, and traditional warning systems can no longer meet the needs of safe driving, mainly due to the following technical defects: Poor scene adaptability and inefficient response mechanism: Existing systems mostly adopt scene recognition methods based on fixed rules or continuous massive data classification. The former relies on static user settings and in-vehicle noise levels, lacking dynamic understanding of driving conditions; the latter requires continuous high-power computing, resulting in high system resource consumption and high response latency, making it impossible to capture sudden high-risk scenarios such as high-speed emergency braking and intersection turns in a timely manner.

[0003] The warning modality is limited and the reliability of warnings is insufficient: Traditional warning systems mostly rely on single auditory or visual cues. When the driver's attention is distracted or a single sensory channel is saturated (such as auditory failure in a noisy environment), critical safety information is easily ignored, and there is a lack of multi-channel redundancy backup mechanisms.

[0004] Insufficient information coordination leads to overload and conflicts: The lack of a unified arbitration mechanism for multi-source information such as in-cabin media, navigation, and safety warnings makes it easy for channel competition to occur when multiple events are triggered simultaneously, resulting in interference with important warnings or excessive warnings causing the driver to "habitually ignore" them.

[0005] The interactive experience is fragmented, and user acceptance is low: existing warnings are mostly isolated alarm events, and their output intensity, modal combination and driving context are not integrated, which can easily cause user resentment and reduce system usability.

[0006] Therefore, there is an urgent need in this field for an event-triggered, multi-source fusion, and multi-modal collaborative intelligent cockpit warning solution to achieve rapid response, accurate prompts, and efficient collaboration in key scenarios, balancing driving safety and user experience. Summary of the Invention

[0007] This application provides a scene-aware intelligent cockpit acoustic control and multimodal prompting method and related system to solve the technical problems of poor scene adaptability, high response latency, insufficient prompting reliability and information overload of traditional warning systems, and achieve the unity of millisecond-level response, high-reliability warning and excellent user experience.

[0008] The technical solution is as follows: Firstly, a method for scene-aware acoustic control and multimodal prompting within an intelligent cockpit is provided, including: Real-time monitoring and identification of predefined trigger events, wherein the trigger events include at least one of vehicle behavior events and external environment events, wherein the vehicle behavior events include at least: sudden braking, sudden acceleration, and turn signal activation accompanied by a steering wheel angle greater than a turning angle threshold; wherein the external environment events include at least: forward accident warning, road hazard warning, and identification of a specific voiceprint; When a triggering event is detected, vehicle dynamic data, external environment data, and driver status data are collected and multi-source data are fused. Based on the fusion results, the current vehicle's driving scenario is identified and output through a large model, and the matching acoustic management strategy and multimodal prompting strategy are called from the preset strategy library based on the driving scenario. When concurrent triggering events occur, the priority of multiple concurrent prompting operations and the allocation of prompting resources in the multimodal prompting strategy are dynamically adjusted according to the risk level of the driving scenario and the priority of the concurrent triggering events. Based on the acoustic management strategy, the sound field reconstruction, audio source volume adjustment and priority control inside and outside the intelligent cockpit are controlled. Based on the adjusted control strategy, the directional output of multimodal collaborative prompt information is controlled. The multimodal collaborative prompt information includes at least one prompting mode among acoustic prompts, visual prompts and tactile prompts, as well as the prompting intensity of the at least one prompting mode.

[0009] In one possible implementation, the system smoothly switches between different alert intensity levels based on changes in the driving scenario or the urgency of the triggering event; and when the triggering event is resolved or the risk is eliminated, the alert process is terminated and the normal cockpit settings are restored.

[0010] In one possible implementation, the multimodal cue intensity levels in the policy library include at least: a basic level with only acoustic modal cueing, an enhanced level with acoustic and optical bimodal cueing, and an emergency level with acoustic, optical, and haptic trimodal cueing; wherein, media pause is included under the emergency level policy.

[0011] In one possible implementation, the priority determination rule for the concurrent triggering events is as follows: emergency safety events have a higher priority than external risk events, and external risk events have a higher priority than driving assistance events; wherein, the emergency safety events include: collision warning, emergency braking; the external risk events include: approach of special vehicles, road hazards; and the driving assistance events include: navigation instructions, blind spot warning.

[0012] In one possible implementation, the control orientation outputs multimodal collaborative prompts, specifically including: mapping the risk orientation to the corresponding physical area of ​​the cabin, outputting warning sounds from the risk direction through multi-channel audio, flashing ambient lights in the corresponding area, activating the vibration motor of the seat on the same side as the risk, and displaying warning icons in the corresponding direction on the HUD.

[0013] Secondly, a scene-aware intelligent cockpit acoustic control and multimodal prompting system is provided, including: The monitoring module is used to monitor and identify predefined trigger events in real time. The trigger events include at least one of vehicle behavior events and external environment events. The vehicle behavior events include at least one of the following: sudden braking, sudden acceleration, and turn signal activation accompanied by a steering wheel angle greater than a turning angle threshold. The external environment events include at least one of the following: forward accident warning, road hazard warning, and identification of a specific voiceprint. The fusion module is used to collect vehicle dynamic data, external environment data, and driver status data, and to perform multi-source data fusion. The calling module is used to identify and output the current vehicle's driving scenario based on the fused results through a large model, and to call the matching acoustic management strategy and multimodal prompting strategy from the preset strategy library based on the driving scenario. The adjustment module is used to dynamically adjust the priority and resource allocation of multiple concurrent prompting operations in the multimodal prompting strategy according to the risk level of the driving scenario and the priority of the concurrent prompting events when concurrent triggering events occur. The control module is used to control the sound field reconstruction, audio source volume adjustment and priority control inside and outside the smart cockpit based on the acoustic management strategy. Based on the adjusted control strategy, it controls the directional output of multimodal collaborative prompt information, wherein the multimodal collaborative prompt information includes at least one prompt modality among acoustic prompts, visual prompts and tactile prompts, and the prompt intensity of the at least one prompt modality.

[0014] Thirdly, an electronic device is provided, comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the methods described above and any possible implementations.

[0015] Fourthly, a computer-readable storage medium is provided, wherein at least one instruction is stored therein, the at least one instruction being loaded and executed by a processor to implement the aspects described above and any possible implementation thereof.

[0016] Fifthly, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the aspects and any possible implementations described above.

[0017] In a sixth aspect, an autonomous vehicle is provided, including the electronic devices described above.

[0018] The beneficial effects of the technical solution provided in this application include at least the following: As can be seen from the above technical solution, the embodiments of this application can monitor and identify predefined triggering events in real time, collect vehicle dynamic data, external environment data, and driver status data, and perform multi-source data fusion; based on the fusion result, the current vehicle driving scenario is identified and output through a large model, and the acoustic management strategy and multimodal prompting strategy matching it are called from the preset strategy library based on the driving scenario; when concurrent triggering events occur, the priority and prompting resource allocation of multiple concurrent prompting operations in the multimodal prompting strategy are dynamically adjusted according to the risk level of the driving scenario and the priority of the concurrent triggering events; based on the acoustic management strategy, the sound field reconstruction, audio source volume adjustment and priority control inside and outside the intelligent cockpit are controlled, and based on the adjusted control strategy, the directional output of multimodal collaborative prompting information is controlled. This application can coordinate media volume, noise reduction mode, and warning sound priority, and simultaneously achieve spatial synchronization and hierarchical collaborative prompting of hearing, vision, and touch, achieving millisecond-level response, improving the reliability of warnings and user acceptance, and achieving the optimal balance between system energy efficiency and safety performance.

[0019] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this application, nor is it intended to limit the scope of this application. Other features of this application will become readily apparent from the following description. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0021] Figure 1 This is a schematic diagram illustrating the steps of a scene-aware intelligent cockpit acoustic control and multimodal prompting method provided in one embodiment of this application.

[0022] Figure 2 This is a schematic diagram of a scene-aware intelligent cockpit acoustic control and multimodal prompting method provided in another embodiment of this application.

[0023] Figure 3 This is a structural block diagram of a scene-aware intelligent cockpit acoustic control and multimodal prompting system provided in another embodiment of this application.

[0024] Figure 4 This is a block diagram of an electronic device used to implement the scene-aware intelligent cockpit acoustic control and multimodal prompting method according to the embodiments of this application. Detailed Implementation

[0025] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These embodiments should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0026] Obviously, the described embodiments are only some, not all, of the embodiments in this application. All other embodiments obtained by those skilled in the art based on the embodiments in this application without inventive effort are within the scope of protection of this application.

[0027] It should be noted that the terminal devices involved in the embodiments of this application may include, but are not limited to, smart devices such as mobile phones, personal digital assistants (PDAs), wireless handheld devices, and tablet computers; the display devices may include, but are not limited to, personal computers, televisions, and other devices with display functions.

[0028] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0029] In view of the technical problems of poor scene adaptability, high response latency, insufficient reliability of prompts, and information overload in existing intelligent cockpit warning systems, this application proposes an acoustic control and multimodal prompting scheme for scene perception in intelligent cockpits. The main inventive concept is as follows: real-time monitoring and identification of predefined triggering events; collection of vehicle dynamic data, external environment data, and driver status data; and multi-source data fusion; based on the fusion result, identification and output of the current vehicle's driving scene through a large model; and based on the driving scene, invocation of matching acoustic management strategies and multimodal prompting strategies from a preset strategy library; when concurrent triggering events occur, dynamic adjustment of the priority and prompting resource allocation of multiple concurrent prompting operations in the multimodal prompting strategy according to the risk level of the driving scene and the priority of the concurrent triggering events; control of sound field reconstruction, audio source volume adjustment, and priority control inside and outside the intelligent cockpit based on the acoustic management strategy; and control of directional output of multimodal collaborative prompting information based on the adjusted control strategy. This application can coordinate media volume, noise reduction mode, and warning tone priority, while simultaneously achieving spatial synchronization and hierarchical collaborative prompts of auditory, visual, and tactile senses. It achieves millisecond-level response, improves warning reliability and user acceptance, and achieves the optimal balance between system energy efficiency and safety performance.

[0030] Reference Figure 1 The diagram shown illustrates the steps of a scene-aware intelligent cockpit acoustic control and multimodal prompting method according to an embodiment of this application. The method may include the following steps: Step 102: Monitor and identify predefined trigger events in real time, wherein the trigger events include at least one of vehicle behavior events and external environment events, the vehicle behavior events include at least: sudden braking, sudden acceleration, and turn signal activation accompanied by a steering wheel angle greater than a turning angle threshold; the external environment events include at least: forward accident warning, road hazard warning, and identification of a specific voiceprint.

[0031] In this application, vehicle dynamic data such as vehicle speed, turn signal status, gear position, steering wheel angle, throttle / brake opening, etc. can be obtained in real time via the CAN bus.

[0032] Among these factors, vehicle speed is the foundation for scene classification and is the primary indicator for judging the macro scene. High speed (>80 km / h) may trigger the "highway" mode; low speed (<20 km / h) and GPS showing a residential area may trigger the "residential area driving" mode; vehicle speed of 0 and gear in P may trigger the "parking" mode.

[0033] The turn signal status is a direct signal of the driver's intention and is one of the most critical behavioral predictive indicators. When the left / right turn signal is activated, the system anticipates that the vehicle is about to change lanes or turn, and at this time it will: a. increase the warning priority of the corresponding side blind spot risk (e.g., enhance the warning of the left horn sound). b. combine with the steering wheel angle to determine whether it is a slight deviation or a large turn.

[0034] The steering wheel angle indicates the vehicle's actual trajectory and, in conjunction with the turn signal, confirms whether the driver's intention is being executed. Specifically, a large steering angle: when confirming the vehicle is turning at an intersection or on a ramp, the system enters a "high alert" mode and enhances external warnings on the steering side. Small angle, high-frequency corrections: when the vehicle is traveling straight on a highway, the system may employ a more balanced warning strategy.

[0035] Throttle / brake opening is a judgment of vehicle dynamics and urgency level. These parameters reflect the driver's aggressiveness and potential risks. Sudden braking indicates a large brake opening; the system will immediately determine this as a high-risk or emergency scenario, potentially triggering the highest level of multimodal warnings (such as forced media suspension, flashing full red ambient lights), and preparing to respond to emergency horns from vehicles behind. Rapid acceleration indicates a large throttle opening, possibly indicating the driver's intention to overtake or pass quickly; the system will adjust its attention to forward and lateral risks accordingly. Smooth throttle / brake operation indicates normal cruising; the system maintains a standard strategy.

[0036] The gear selection confirms the vehicle's operating mode, used to acknowledge changes in the driving scenario. D (Drive): Normal driving mode. R (Reverse): The system switches to a "reverse" mode, potentially suppressing rear horn noise (because vehicles behind are usually waiting, not warning you), while focusing alert resources on collision warnings from the reversing radar / camera. P (Park): Indicates a parked state, disabling or significantly reducing external warning enhancements.

[0037] Simultaneously, external environmental data can be acquired through external sensors. For example, road type (highway, urban area, back alley) and real-time traffic conditions can be obtained through GPS / navigation systems; weather and lighting conditions can be obtained through light / rain sensors. Driver status data can be monitored through in-vehicle cameras or infrared sensors to detect the driver's head orientation and eyelid opening, providing a preliminary assessment of their level of concentration.

[0038] Step 104: When a triggering event is detected, collect vehicle dynamic data, external environment data, and driver status data, and perform multi-source data fusion.

[0039] Step 106: Based on the fusion results, identify and output the current vehicle's driving scenario through the large model, and call the matching acoustic management strategy and multimodal prompting strategy from the preset strategy library based on the driving scenario.

[0040] Based on the fused data, the system uses preset rules or machine learning models to identify and output the current driving scenario in real time, such as "smooth highway", "congested city", "highway at night in rain", "turning at intersection", etc.

[0041] Additionally, a configurable strategy library can be embedded to match an optimal acoustic management and multimodal prompting strategy for each driving scenario.

[0042] The system deconstructs the continuous driving process into discrete scenarios with distinct characteristics, and tailors acoustic management and multimodal prompting strategies for each scenario. For example: Scenario 1: Highway Cruise Strategy: Keep media volume at a moderate level, turn on active noise cancellation to counteract road noise, amplify all vehicle horn sounds, and provide strong audible, visual (HUD), and tactile (seat vibration) cues for sudden horn honking.

[0043] Scenario 2: Following another car in traffic congestion in the city Strategy: The media volume is automatically reduced, and the horn sounds at close range and high risk (such as the horn of a vehicle braking suddenly from behind) are selectively amplified. Ordinary reminder horns are only slightly amplified or ignored to avoid frequent interference.

[0044] Step 108: When a concurrent triggering event occurs, dynamically adjust the priority of multiple concurrent prompting operations and the allocation of prompting resources in the multimodal prompting strategy according to the risk level of the driving scenario and the priority of the concurrent triggering event.

[0045] In one possible implementation, the priority determination rule for the concurrent triggering events is as follows: emergency safety events have a higher priority than external risk events, and external risk events have a higher priority than driving assistance events; wherein, the emergency safety events include: collision warning, emergency braking; the external risk events include: approach of special vehicles, road hazards; and the driving assistance events include: navigation instructions, blind spot warning.

[0046] In one possible implementation, the multimodal cue intensity levels in the policy library include at least: a basic level with only acoustic modal cueing, an enhanced level with acoustic and optical bimodal cueing, and an emergency level with acoustic, optical, and haptic trimodal cueing; wherein, media pause is included under the emergency level policy.

[0047] As can be seen, this application defines a "level of intensity" for the prompts. When the preferred mode may fail, the system will automatically activate a stronger or backup mode, forming a layered and complementary prompting system. The prompt level will be automatically upgraded based on the severity of the scenario and the driver's status (e.g., detecting that the driver is not looking at the road).

[0048] Step 110: Based on the acoustic management strategy, control the sound field reconstruction, audio source volume adjustment and priority control inside and outside the intelligent cockpit. Based on the adjusted control strategy, control the directional output of multimodal collaborative prompt information, wherein the multimodal collaborative prompt information includes at least one prompt modality among acoustic prompts, visual prompts and tactile prompts, and the prompt intensity of the at least one prompt modality.

[0049] Optionally, in this application, the system can also smoothly switch between different alert intensity levels based on changes in the driving scenario or the urgency of the triggered event; and terminate the alert process and restore normal cockpit settings when the triggered event is resolved or the risk is eliminated.

[0050] In one possible implementation, the control orientation outputs multimodal collaborative prompts, specifically including: mapping the risk orientation to the corresponding physical area of ​​the cabin, outputting warning sounds from the risk direction through multi-channel audio, flashing ambient lights in the corresponding area, activating the vibration motor of the seat on the same side as the risk, and displaying warning icons in the corresponding direction on the HUD.

[0051] Reference Figure 2 The diagram shown is a flowchart illustrating the scene-aware acoustic control and multimodal prompting method for intelligent cockpits provided in this embodiment of the application.

[0052] Step 202: Event Monitoring and Triggering The system is normally in a low-power monitoring state, monitoring predefined high-risk events in real time. When any trigger condition such as emergency braking, V2X emergency message reception, or specific voiceprint recognition is detected, step 204 is executed immediately.

[0053] Step 204: Multi-source information fusion Simultaneously collect vehicle dynamic data, external environment data, driver status data, and V2X communication data, perform data cleaning and format standardization, and form a unified scene perception dataset to provide comprehensive input for scene analysis.

[0054] Step 206: Scene Analysis and Strategy Invocation Analyze the fused dataset and match semantic scene labels: Example 1: Emergency braking event + vehicle speed 80km / h + GPS road type = highway → scene label "highway emergency braking"; Example 2: left turn signal activated + steering wheel angle > 45° + GPS location = intersection → scene label "intersection left turn"; Based on the scene label, call the corresponding combination strategy from the following strategy library, including acoustic management parameters and multimodal prompt level.

[0055] Strategy library Step 208: Resource Arbitration and Warning Enforcement Based on the scenario's risk level and the priority of concurrent events, perform the following operations: Acoustic Arbitration: Adjust media volume (e.g., pause media in emergency mode), switch active noise cancellation mode (e.g., suppress rain noise), enhance external warning sounds, and achieve directional output through sound field reconstruction; Multimodal collaboration: Activate corresponding modal combinations according to the prompt level: Basic level: only enhance external warning sounds; Enhanced level: acoustic enhancement + HUD corresponding directional icons + ambient light flashing; Emergency level: acoustic enhancement + visual cues + vibration of the seat / steering wheel on the same side of the risk + media pause; Orientation synchronization: map the risk location (such as the left rear, the intersection in front) to the physical area of ​​the cabin to achieve spatial synchronization of sound, light and touch prompts.

[0056] Step 210: Dynamic Switching and Termination When the risk level of a scenario changes (e.g., risk decreases) or a new event is triggered, the control system smoothly switches between different levels of multimodal prompts; when the triggered event is resolved (e.g., brake opening returns to normal, V2X risk is resolved), the system gradually restores the normal cockpit settings and terminates the warning process.

[0057] Compared to existing technologies, this invention offers the following significant advantages: Millisecond-level response speed: Employing an event-triggered mechanism instead of continuous monitoring, the system instantly activates from a low-power standby state, achieving a critical scene recognition and decision-making delay of ≤50ms, thus solving the problem of lag in traditional systems; Significantly improved warning reliability: Constructing an "auditory-visual-tactile" multimodal redundant system automatically activates backup channels when a single modality fails, reducing the information false alarm rate by over 90% and ensuring timely warnings even in complex environments; Precise scene adaptability: Multi-source information fusion enables dynamic scene analysis, and strategy library mapping ensures a high degree of matching between warning intensity, modal combinations, and driving context, avoiding both "disturbing the peace" and "insufficient warnings." The system addresses the contradictions in information sharing and collaboration: a centralized resource arbitration mechanism resolves conflicts arising from competition among multiple information sources, dynamically allocates channel resources according to priority, and fundamentally solves the problem of information overload; optimal system energy efficiency: an event-triggered working mode significantly reduces system power consumption, and high-risk scenarios instantly activate all-dimensional resources, achieving global optimization of safety and energy efficiency; and superior user experience: multimodal prompts are deeply integrated with driving scenarios, and directional and contextualized interaction methods avoid isolated alarms, improving system availability and user acceptance.

[0058] Example 1: High-speed emergency braking scenario Event Triggering: The vehicle is traveling on a highway (100km / h) when the driver suddenly brakes, reaching 85% braking distance. Event Triggering Module 1 detects the "emergency braking event" and immediately activates the system. Multi-Source Information Fusion: Cockpit Domain Controller 2 collects data: vehicle speed 100km / h, braking distance 85%, road type = highway, no other messages received via V2X, driver's head facing the road. Scene Analysis and Strategy Invocation: Scene Analysis Engine 22 outputs the label "High-Speed ​​Emergency Braking," and Strategy Decision Engine 23 invokes the corresponding strategy: media pause, active noise cancellation to suppress road noise, enhanced external warning sound, and emergency-level multimodal prompts. Warning Execution: Acoustic: Media immediately pauses, active noise cancellation switches to "road noise suppression" mode, and the audio system creates a sense of urgency. Visual: The HUD displays a full-width red braking warning icon, and the ambient light on the upper edge of the rear window flashes red rapidly (frequency 5Hz). Tactile: The driver's seat and steering wheel simultaneously initiate high-frequency vibration (150Hz). External Communication: "Emergency Braking" is broadcast to vehicles within 500m behind via V2X. Warning message; Termination: After 3 seconds, the brake opening returns to <30%, the system gradually stops vibrating and the ambient light flashing, media playback resumes (volume gradually increases from 0 to 50% of the original volume), and the warning process terminates.

[0059] Example 2: Left turn at an intersection Event Trigger: The vehicle is at an intersection in the city (speed 30km / h). The driver activates the left turn signal, and the steering wheel angle reaches 50°. Event trigger module 1 detects the "sudden turn + turn signal activation" event and activates the system. Multi-source information fusion: Data collected: vehicle speed 30km / h, left turn signal activation, steering wheel angle 50°, GPS location = intersection, left microphone detects slight horn honking. Scene analysis and strategy invocation: Output label "left turn at intersection", invoke strategy: media volume reduced by 30%, enhance left / opposite external warning sound, enhanced multimodal prompts. Warning execution: Acoustic: media volume reduced from 60% to 42%, left audio channel enhances external horn volume (increases by 15dB); Visual: red flashing horn icon displayed on the left front of the HUD, left door ambient light flashes red (frequency 2Hz); Tactile: vibration motor on the left side of the driver's seat starts (100Hz, continuous for 1 second). (seconds); Switch: The vehicle completes a left turn (steering wheel angle < 10°), the scene label is updated to "Normal Driving in Urban Area", the system switches to basic level prompts, the media volume returns to its original level, and visual and tactile prompts are terminated.

[0060] Performance Verification: Through 1000 simulated scenario tests on 10 test vehicles (including fuel vehicles and new energy vehicles), the results are as follows: Key scenario response latency: average 38ms, meeting millisecond-level requirements; Warning recognition accuracy: driver's accuracy in identifying the location and nature of risks is ≥96%, an improvement of 35% compared to the traditional single-modality system; User acceptance: 85% of testers believe that the multimodal prompts are "not disturbing and effective," with no "habitual ignoring" phenomenon; System power consumption: power consumption ≤5W under normal conditions, peak power consumption ≤20W after triggering, a reduction of 60% compared to the continuous monitoring solution.

[0061] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0062] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0063] Figure 3 This illustration shows a structural block diagram of a scene-aware intelligent cockpit acoustic control and multimodal prompting system according to an embodiment of this application. Figure 3As shown. The scene-aware intelligent cockpit acoustic control and multimodal prompting system 300 of this embodiment may include a monitoring module 301, a fusion module 302, a calling module 303, an adjustment module 304, and a control module 305. The monitoring module 301 is used to monitor and identify predefined trigger events in real time. The trigger events include at least one of vehicle behavior events and external environment events. The vehicle behavior events include at least: sudden braking, sudden acceleration, and turn signal activation accompanied by a steering wheel angle greater than a turning angle threshold. The external environment events include at least: forward accident warning, road hazard warning, and identification of a specific voiceprint. The fusion module 302 is used to collect vehicle dynamic data, external environment data, and driver state data, and perform multi-source data fusion. The calling module 303 is used to identify and output the current vehicle's driving scenario based on the fused results using a large model, and to call matching acoustic management strategies and multimodal prompting strategies from a preset strategy library based on the driving scenario. The adjustment module 304 is used to dynamically adjust the priority and resource allocation of multiple concurrent prompting operations in the multimodal prompting strategy according to the risk level of the driving scenario and the priority of the concurrent triggering events when concurrent triggering events occur. The control module 305 is used to control the sound field reconstruction, audio source volume adjustment, and priority control inside and outside the intelligent cockpit based on the acoustic management strategy. Based on the adjusted control strategy, it controls the directional output of multimodal collaborative prompting information, wherein the multimodal collaborative prompting information includes at least one prompting modality among acoustic prompts, visual prompts, and tactile prompts, and the prompting intensity of the at least one prompting modality.

[0064] It should be noted that some or all of the device for scene-aware intelligent cockpit acoustic control and multimodal prompting in this embodiment can be an application located on a local terminal, or it can be a plugin or software development kit (SDK) or other functional unit set in an application located on a local terminal, or it can be a processing engine located on a network-side server, or it can be a distributed system located on the network side, such as a processing engine or distributed system in a network-side autonomous driving platform, etc. This embodiment does not impose any particular limitations on this.

[0065] It is understood that the application can be a native program installed on the local terminal, or it can be a web application of a browser on the local terminal. This embodiment does not limit this.

[0066] Optionally, in one possible implementation of this embodiment, it further includes: a switching module, used to smoothly switch between different prompt intensity levels according to changes in the driving scenario or the urgency of the triggering event; and to terminate the prompting process and restore normal cockpit settings when the triggering event is resolved or the risk is eliminated.

[0067] Optionally, in one possible implementation of this embodiment, the multimodal cue intensity levels in the strategy library include at least: a basic level with only acoustic modal cueing, an enhanced level with dual acoustic and optical cueing, and an emergency level with trimodal acoustic, optical, and haptic cueing; wherein, media pause is included under the emergency level strategy.

[0068] Optionally, in one possible implementation of this embodiment, the priority determination rule for concurrent triggering events is as follows: emergency safety events have a higher priority than external risk events, and external risk events have a higher priority than driving assistance events; wherein, the emergency safety events include: collision warning, emergency braking; the external risk events include: special vehicle approach, road hazard; and the driving assistance events include: navigation instructions, blind spot warning.

[0069] Optionally, in one possible implementation of this embodiment, when the control module 305 outputs multimodal collaborative prompt information for directional control, it is specifically used to map the risk location to the corresponding physical area of ​​the cabin, output warning sounds from the risk direction through multi-channel audio, flash the ambient light in the corresponding area, start the vibration motor of the seat on the same side as the risk, and display a warning icon in the corresponding direction of the HUD.

[0070] In this embodiment, predefined triggering events can be monitored and identified in real time, vehicle dynamic data, external environment data, and driver status data can be collected, and multi-source data fusion can be performed. Based on the fusion result, the current vehicle driving scenario is identified and output through a large model, and the acoustic management strategy and multimodal prompting strategy matching the driving scenario are called from the preset strategy library. When concurrent triggering events occur, the priority and prompting resource allocation of multiple concurrent prompting operations in the multimodal prompting strategy are dynamically adjusted according to the risk level of the driving scenario and the priority of the concurrent triggering events. Based on the acoustic management strategy, the sound field reconstruction, audio source volume adjustment, and priority control inside and outside the smart cockpit are controlled. Based on the adjusted control strategy, the directional output of multimodal collaborative prompting information is controlled. This application can coordinate media volume, noise reduction mode, and warning sound priority, and simultaneously achieve spatial synchronization and hierarchical collaborative prompting of hearing, vision, and touch. It achieves millisecond-level response, improves the reliability of warnings and user acceptance, and achieves the optimal balance between system energy efficiency and safety performance.

[0071] One embodiment of this application provides a computer-readable storage medium storing at least one instruction, which is loaded and executed by a processor to implement the method for scene-aware intelligent cockpit acoustic control and multimodal prompting as described above.

[0072] One embodiment of this application provides an electronic device, which includes a processor and a memory. The memory stores at least one instruction, which is loaded and executed by the processor to implement the scene-aware intelligent cockpit acoustic control and multimodal prompting method described above.

[0073] One embodiment of this application provides an autonomous driving vehicle, including the electronic devices described above. Specifically, the autonomous driving vehicle can be a Level 2 or higher vehicle.

[0074] The collection, storage, use, processing, transmission, provision, and disclosure of user personal information involved in the technical solution of this application all comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0075] Figure 4 A schematic block diagram of an example electronic device 400 that can be used to implement embodiments of this application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the application described and / or claimed herein.

[0076] like Figure 4 As shown, the electronic device 400 includes a computing unit 401, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 402 or a computer program loaded from a storage unit 408 into a random access memory (RAM) 403. The RAM 403 may also store various programs and data required for the operation of the electronic device 400. The computing unit 401, ROM 402, and RAM 403 are interconnected via a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0077] Multiple components in electronic device 400 are connected to I / O interface 405, including: input unit 406, such as keyboard, mouse, etc.; output unit 407, such as various types of displays, speakers, etc.; storage unit 408, such as disk, optical disk, etc.; and communication unit 409, such as network card, modem, wireless transceiver, etc. Communication unit 409 allows electronic device 400 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0078] The computing unit 401 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 401 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 401 performs the various methods and processes described above, such as the method for scene-aware in-cabin acoustic control and multimodal prompting. For example, in some embodiments, the method for scene-aware in-cabin acoustic control and multimodal prompting can be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 408. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 400 via ROM 402 and / or communication unit 409. When the computer program is loaded into RAM 403 and executed by the computing unit 401, one or more steps of the method for scene-aware in-cabin acoustic control and multimodal prompting described above can be performed. Alternatively, in other embodiments, computing unit 401 may be configured by any other suitable means (e.g., by means of firmware) to perform a method for scene-aware intelligent cockpit acoustic control and multimodal cues.

[0079] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, at least one input device, and at least one output device.

[0080] The program code used to implement the methods of this application may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing device, such that when executed by the processor or controller, the functions / operations specified in the flowcharts and / or block diagrams are implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0081] In the context of this application, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. Machine-readable media can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0082] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0083] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0084] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0085] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0086] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for scene-aware acoustic control and multimodal prompting in an intelligent cockpit, characterized in that, include: Real-time monitoring and identification of predefined trigger events, wherein the trigger events include at least one of vehicle behavior events and external environment events, wherein the vehicle behavior events include at least: sudden braking, sudden acceleration, and turn signal activation accompanied by a steering wheel angle greater than a turning angle threshold; wherein the external environment events include at least: forward accident warning, road hazard warning, and identification of a specific voiceprint; Collect vehicle dynamic data, external environment data, and driver status data, and perform multi-source data fusion; Based on the fusion results, the current vehicle's driving scenario is identified and output through a large model, and the matching acoustic management strategy and multimodal prompting strategy are called from the preset strategy library based on the driving scenario. When concurrent triggering events occur, the priority of multiple concurrent prompting operations and the allocation of prompting resources in the multimodal prompting strategy are dynamically adjusted according to the risk level of the driving scenario and the priority of the concurrent triggering events. Based on the acoustic management strategy, the sound field reconstruction, audio source volume adjustment and priority control inside and outside the intelligent cockpit are controlled. Based on the adjusted control strategy, the directional output of multimodal collaborative prompt information is controlled. The multimodal collaborative prompt information includes at least one prompting mode among acoustic prompts, visual prompts and tactile prompts, as well as the prompting intensity of the at least one prompting mode.

2. The method as described in claim 1, characterized in that, The method further includes: Smoothly switch between different levels of alert intensity based on changes in the driving scenario or the urgency of the triggering event; and When the triggered event is resolved or the risk is eliminated, the prompting process terminates and the normal cockpit settings are restored.

3. The method as described in claim 1, characterized in that, The multimodal cue strength levels in the policy library include at least: The system includes a basic level with only acoustic modal cues, an enhanced level with dual acoustic and optical cues, and an emergency level with trimodal cues including acoustic, optical, and haptic cues; among these, the emergency level strategy includes media pause.

4. The method as described in claim 3, characterized in that, The priority determination rule for concurrent triggering events is as follows: emergency safety events have a higher priority than external risk events, and external risk events have a higher priority than driving assistance events. The emergency safety events include: collision warning and emergency braking; the external risk events include: approach of special vehicles and road hazards; and the driving assistance events include: navigation instructions and blind spot warnings.

5. The method as described in claim 3, characterized in that, Control the orientation output of multimodal collaborative prompts, specifically including: The risk location is mapped to the corresponding physical area in the cabin. Warning sounds are output from the direction of the risk through multi-channel audio, ambient lights in the corresponding area flash, the vibration motor of the seat on the same side as the risk is activated, and a warning icon is displayed in the corresponding direction on the HUD.

6. A scene-aware intelligent cockpit acoustic control and multimodal prompting system, characterized in that, include: The monitoring module is used to monitor and identify predefined trigger events in real time. The trigger events include at least one of vehicle behavior events and external environment events. The vehicle behavior events include at least one of the following: sudden braking, sudden acceleration, and turn signal activation accompanied by a steering wheel angle greater than a turning angle threshold. The external environment events include at least one of the following: forward accident warning, road hazard warning, and identification of a specific voiceprint. The fusion module is used to collect vehicle dynamic data, external environment data, and driver status data, and to perform multi-source data fusion. The calling module is used to identify and output the current vehicle's driving scenario based on the fused results through a large model, and to call the matching acoustic management strategy and multimodal prompting strategy from the preset strategy library based on the driving scenario. The adjustment module is used to dynamically adjust the priority and resource allocation of multiple concurrent prompting operations in the multimodal prompting strategy according to the risk level of the driving scenario and the priority of the concurrent prompting events when concurrent triggering events occur. The control module is used to control the sound field reconstruction, audio source volume adjustment and priority control inside and outside the smart cockpit based on the acoustic management strategy. Based on the adjusted control strategy, it controls the directional output of multimodal collaborative prompt information, wherein the multimodal collaborative prompt information includes at least one prompt modality among acoustic prompts, visual prompts and tactile prompts, and the prompt intensity of the at least one prompt modality.

7. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1-5.

8. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-5.

9. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-5.

10. An autonomous vehicle, including the electronic equipment as claimed in claim 7.

Images