Vehicle-mounted operation guidance method and vehicle

Abstract

This application discloses an in-vehicle operation guidance method and vehicle, applied in the field of smart cockpit technology. The method identifies at least two guidance channels with different sensory modalities as target channels from multiple guidance channels, and configures the visual guidance channel to present the guidance content visually in a position that does not overlap with the driver's current primary field of vision. This allows the user to receive visual guidance information without shifting their gaze, while guidance channels with other sensory modalities further assist in information transmission. As a result, the driver's gaze remains consistently focused on the primary field of vision ahead, reducing safety hazards caused by the driver's gaze leaving their primary field of vision. Furthermore, guidance information can be efficiently received through multimodal collaboration, improving the efficiency of guidance information transmission while ensuring driving safety.

Description

Technical Field

[0001] This application relates to the field of intelligent cockpit technology, specifically to an in-vehicle operation guidance method and a vehicle. Background Technology

[0002] As vehicles become increasingly intelligent, the number of applications and the complexity of functions in in-vehicle systems are constantly growing. When users first encounter or use a specific function, they often need to rely on a guidance system to learn the operating steps. Related technologies commonly employ screen-based graphic or video guidance solutions, such as displaying guidance prompts on the vehicle's screen to guide users through the steps. These solutions typically use a passive response mode that triggers the guidance; users need to actively read or observe the guidance content on the screen and switch between operation and eye contact. Because the output of guidance information depends entirely on the user's active attention and gaze shift, a time lag exists between guidance and operation. Drivers need to frequently divert their attention to understand instructions while driving, which poses certain safety hazards and operational efficiency bottlenecks in the driving environment.

[0003] Therefore, how to reduce the interference of guidance information on the driver and ensure driving safety has become an urgent technical problem to be solved in the field of vehicle human-machine interaction technology. Summary of the Invention

[0004] To address the aforementioned technical problems, this application is proposed. Embodiments of this application provide an in-vehicle operation guidance method and vehicle. Through multimodal collaborative presentation, users can receive directional prompts through multiple sensory modalities without actively searching for guidance information. In particular, by placing visual guidance in non-primary field of vision areas, it effectively reduces user eye shifts, thereby achieving efficient transmission of guidance information while reducing driving safety risks.

[0005] According to a first aspect of this application, an in-vehicle operation guidance method is provided, comprising: in response to an operation guidance requirement, acquiring a corresponding guidance request; based on the guidance request, determining at least two guidance channels of different sensory modalities from a plurality of guidance channels as target channels; wherein, if the target channels include a visual guidance channel, the visual guidance channel is configured to: visually present guidance content at a position that does not overlap with the driver's current main field of vision; generate multimodal guidance content that matches the guidance request; and allocate the multimodal guidance content to the target channels to collaboratively present operation guidance information.

[0006] As one possible implementation, determining at least two different sensory modal guidance channels as target channels from multiple guidance channels based on the guidance request includes: responding to the guidance request, analyzing the functional structure and operational complexity of the target application to generate multiple task nodes, each task node corresponding to an operation step; determining the task dependencies between the multiple task nodes; wherein the task dependencies characterize the execution order between the task nodes; evaluating a difficulty coefficient for each task node; wherein the difficulty coefficient characterizes the operational complexity of the task node; assigning at least one recommended guidance channel combination to each task node based on the difficulty coefficient of each task node; and determining the target channel based on at least one recommended guidance channel combination.

[0007] By breaking down the guidance request into multiple dependent task nodes and evaluating the difficulty coefficient of each node and assigning a recommended combination of guidance channels, the guidance system can accurately match appropriate sensory modal guidance for each step based on the complexity and logical sequence of the operation steps. This improves the matching degree between guidance content and operation process, while also enhancing the accuracy and efficiency of guidance.

[0008] As one possible implementation, before allocating the multimodal guidance content to the target channel to collaboratively present operation guidance information, the method further includes: analyzing the user's current action trends and / or operation history to identify the user's operation intention and determine the user's next target operation object; wherein, allocating the multimodal guidance content to the target channel to collaboratively present operation guidance information includes: in response to the user's operation intention, allocating the multimodal guidance content to the target channel and initiating output before the user actually contacts the target operation object.

[0009] Before the user actually interacts with the target object, the system analyzes the user's current action trends and / or operation history to identify their operational intentions, and pre-allocates multimodal guidance content to the target channel and initiates output. This shortens the time difference between guidance information and the user's actual operation, improves the initiative and timeliness of guidance response, avoids operation pauses that may occur due to passive waiting, and avoids distraction caused by the user actively recalling or searching for operation steps while driving, thus improving driving safety.

[0010] As one possible implementation, the method further includes: acquiring historical interaction behavior data associated with the current user; calculating the current user's operation proficiency index based on the historical interaction behavior data; wherein the operation proficiency index is calculated based on the operation success rate, the ratio of operation completion time to target time, and the error rate; and adjusting the level of detail of the multimodal guidance content or the combination of guidance channels based on the operation proficiency index.

[0011] By acquiring users' historical interaction data and calculating operation proficiency indicators based on operation success rate, completion time ratio, and error rate, the level of detail of multimodal guidance content or the combination of guidance channels can be dynamically adjusted. This allows guidance to automatically adapt to the user's current ability level, thereby reducing the probability of insufficient information for novice users and redundant content for experienced users in a unified guidance mode. It also improves the matching degree between guidance content and users' actual needs, while enhancing the guidance efficiency and experience satisfaction of different user groups.

[0012] As one possible implementation, adjusting the level of detail of the multimodal guidance content or the combination of guidance channels based on the operation proficiency index includes: simplifying the multimodal guidance content and / or reducing the number of target channels when the operation proficiency index is higher than a first threshold; increasing the multimodal guidance content, increasing the number of guidance repetitions, or enabling more guidance channels when the operation proficiency index is lower than a second threshold; wherein the first threshold is greater than the second threshold.

[0013] By setting clear first and second thresholds, users' operational proficiency is divided into three levels, and clear and definite adjustment actions are performed for each level. This reduces the ambiguity and uncertainty of the guidance strategy adjustment, improves the accuracy and stability of personalized guidance control, and enables novice users to receive more ample multimodal guidance support, while experienced users receive simpler and more efficient guidance. While improving the rationality of guidance resource allocation, it also ensures the smoothness of user operation.

[0014] As one possible implementation, the method further includes: acquiring scene parameters characterizing the current driving scenario; wherein the scene parameters include at least one of vehicle speed, ambient light intensity, and driver fatigue state; and enabling, disabling, or adjusting the operating parameters of the guidance channel in real time based on preset scene guidance strategy mapping rules and the scene parameters.

[0015] By acquiring scene parameters such as vehicle speed, ambient light intensity, and driver fatigue status in real time, and dynamically enabling, disabling, or adjusting the operating parameters of the guidance channel according to preset rules, the guidance can proactively adapt to the ever-changing driving environment. This reduces the risk of interference or failure that may occur in fixed guidance modes under special scenarios, and improves the adaptability and safety of guidance behavior in different driving scenarios.

[0016] As one possible implementation, the operation parameters of the guidance channel are enabled, disabled, or adjusted in real time based on the preset scene guidance strategy mapping rules and the scene parameters, including: disabling the visual guidance channel when the vehicle speed is greater than or equal to a preset safety threshold; adjusting the brightness and contrast parameters of the visual guidance channel display content when the ambient light intensity exceeds a preset range; and simplifying the structure of the multimodal guidance content when driver fatigue is detected.

[0017] For the three key scenario parameters of vehicle speed, lighting conditions, and driver fatigue, specific adjustment actions were set, thereby reducing the possibility of the system responding vaguely when facing high safety risks or environmental interference, improving the certainty and effectiveness of the guidance strategy in key scenarios, reducing the adverse effects of visual distraction, visual blur or cognitive overload on driving safety, and improving the stability of the system under complex and dangerous driving conditions.

[0018] As one possible implementation, the step of responding to an operation guidance request and obtaining the corresponding guidance request includes: when multiple applications generate operation guidance requests simultaneously, assigning a priority attribute to each corresponding guidance request; wherein the priority attribute includes the application type and / or the urgency of the operation; based on a preset priority arbitration mechanism, allowing the highest priority guidance request to occupy the guidance channel according to priority order, and placing other guidance requests in a waiting queue or directly rejecting them.

[0019] When multiple applications generate operation guidance requests simultaneously, a priority arbitration mechanism is used to allow the highest priority request to occupy the guidance channel, while other requests are queued or rejected in order. This reduces the possibility of information overlap, confusion, or loss of key guidance information caused by multi-application guidance conflicts, improves the priority delivery capability of emergency or security-related guidance, and ensures that the guidance of core functions is not delayed or interrupted by other low-priority requests.

[0020] As one possible implementation, generating multimodal guidance content that matches the guidance request further includes: generating at least one of the following guidance sub-contents: generating voice guidance content; wherein the voice guidance content includes natural language instructions for instructing the user to perform corresponding operation steps; generating haptic guidance content; wherein the haptic guidance content includes vibration patterns with directionality or functional relevance; generating visual guidance content; wherein the visual guidance content includes at least one of lines, arrows, outline highlighting, or operation trajectory animation, for matching with physical control components inside the vehicle in spatial coordinates and superimposing them on the physical control components, which are located in the driver's field of vision observed through the non-overlapping display area; generating gesture guidance content; wherein the gesture guidance content includes a three-dimensional gesture trajectory animation for demonstrating the operation method.

[0021] By providing clear content generation methods for modalities such as voice, touch, vision, and gesture, the probability of non-standard and unintuitive guidance content is reduced, the ability of users to quickly and accurately understand operation instructions through multiple sensory channels is improved, the burden of users switching between screen and physical components is reduced, and the intuitiveness and operational efficiency of guidance are enhanced.

[0022] According to a second aspect of this application, a vehicle is provided, comprising: a guidance channel including a visual guidance channel, a tactile guidance channel, and a voice guidance channel; an on-board sensor unit for collecting scene parameters; and an on-board operation guide communicatively connected to the guidance channel and the on-board sensor unit, the on-board operation guide being used to execute the on-board operation guidance method as described in the first aspect or any implementation thereof.

[0023] According to a third aspect of this application, an in-vehicle operation guidance device is provided, comprising: an acquisition module for acquiring a corresponding guidance request in response to an operation guidance requirement; a channel determination module for determining at least two different sensory modal guidance channels as target channels from a plurality of guidance channels based on the guidance request; wherein, if the target channels include a visual guidance channel, the visual guidance channel is configured to: visually present guidance content at a position that does not overlap with the driver's current main field of vision; a content generation module for generating multimodal guidance content that matches the guidance request; and a guidance execution module for allocating the multimodal guidance content to the target channels to collaboratively present operation guidance information.

[0024] According to a fourth aspect of this application, a computer device is provided, the computer device comprising: one or more processors; a memory; and one or more application programs, wherein the one or more application programs are stored in the memory and configured to be executed by the processor to implement the method as described in the first aspect or any implementation thereof.

[0025] According to a fifth aspect of this application, a computer-readable storage medium is provided, the storage medium storing a computer program for performing the method as described in the first aspect or any implementation thereof.

[0026] According to a sixth aspect of this application, an electronic device is provided, including a module for performing the method as described in the first aspect or any implementation thereof.

[0027] According to a seventh aspect of this application, a computer program product is provided, comprising program code for performing the method as described in the first aspect or any implementation thereof.

[0028] The in-vehicle operation guidance method and vehicle provided in this application, firstly, when a user needs to use a certain function, a guidance request is obtained based on the user's operation guidance needs. Subsequent guidance actions provide a clear starting point, avoiding blind activation or omission of guidance functions and ensuring the timeliness and relevance of guidance. Secondly, at least two different sensory modal channels are activated, diverting guidance information to multiple parallel sensory pathways and reducing over-reliance on a single visual channel. For example, visual guidance content is projected into an area that does not overlap with the driver's primary field of vision, so that the driver does not need to take their eyes off the road when receiving visual guidance information, reducing the safety risks caused by gaze deviation. At the same time, the addition of non-visual channels allows the driver to obtain guidance information through hearing or tactile sensation even without taking their eyes off the road, further reducing cognitive load. Then, corresponding content is generated for each sensory channel based on the guidance request, making the information of each channel semantically consistent and functionally complementary. This avoids contradictory or redundant information output by different channels, ensuring the accuracy of guidance and the consistency of user experience. Finally, by distributing the generated multimodal content to their respective corresponding channels, the driver can obtain complete operation guidance simultaneously through multimodal senses. This not only improves the efficiency of guiding information transmission, but also enables drivers to efficiently and safely learn how to operate in-vehicle applications while maintaining focus on driving tasks. Attached Figure Description

[0029] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0030] Figure 1 This is a flowchart illustrating an exemplary embodiment of the in-vehicle operation guidance method provided in this application.

[0031] Figure 2 This is a schematic diagram of the structure of an on-board operation guidance device provided in an exemplary embodiment of this application.

[0032] Figure 3 This is a structural diagram of an electronic device provided in an exemplary embodiment of this application. Detailed Implementation

[0033] Hereinafter, exemplary embodiments according to this application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein.

[0034] In the field of in-vehicle human-machine interaction guidance technology, to enable drivers and passengers to quickly master the use of various in-vehicle applications, the currently prevalent technical solution is a screen guidance method based on a single visual channel. For example, after a user launches an application or triggers a specific function, the system presents preset guidance information in the form of static or dynamic images on a fixed display area. Through a guidance page containing graphic and textual prompts, it instructs the driver to perform the corresponding operation steps. This achieves basic operation guidance functions at a relatively low cost and is a direct means of solving the problem of users being unfamiliar with application operation procedures.

[0035] However, this solution presents significant safety hazards when applied to real-world driving environments—dynamic scenarios with extremely high safety requirements. This is because, to achieve clear visual information presentation, the solution requires users to shift their gaze from the road ahead to a fixed screen to read and understand textual instructions. However, a fundamental requirement for safe driving is that the driver's gaze should primarily remain on the road ahead to continuously perceive changes in the traffic environment. This conflict creates safety risks during driving. For example, in scenarios where the driver needs to simultaneously process navigation information and adjust the entertainment system volume, because guidance information can only be displayed in a single visual modality on a fixed screen area, the driver must frequently switch their gaze between observing the road and reading screen instructions. This gaze shift, especially in complex traffic conditions, directly disrupts the driver's perception of the road ahead, increasing the risk of traffic accidents.

[0036] The conflict arises because a single visual guidance channel requires all information to be transmitted through the human eye's visual pathway. However, the driver's primary visual load during driving is already occupied with observing the road and environment ahead, leading to severe bandwidth resource competition within the visual channel itself. Existing screen display positions do not overlap with the driver's primary field of vision, forcing a physical eye-shifting path and lengthening the information acquisition path. Furthermore, when first using complex applications, users need to match abstract icons or text descriptions on the screen with their physical control components (such as knobs and buttons). This process involves multiple transitions in spatial positioning and functional cognition, further increasing the time spent on distraction and the burden on working memory. Moreover, existing technologies lack the ability to anticipate and integrate the collaborative work of different sensory modalities. When the driving environment changes (such as strong light causing screen glare), this single-modal guidance scheme completely fails, lacking any redundant or alternative information transmission strategies. These factors collectively limit the ability of existing solutions to balance driving safety and efficient guidance.

[0037] To overcome the aforementioned conflicts, this application proposes an in-vehicle operation guidance method and vehicle. It utilizes at least two guidance channels with different sensory modalities to collaboratively present guidance information. When the target channel includes a visual guidance channel, the visual guidance content is projected onto a display area that does not overlap with the driver's primary field of vision. By diverting guidance information to non-visual channels such as auditory and tactile senses, and positioning visual indicators in the head-up display area of ​​the road ahead, the driver can receive guidance without taking their eyes off the road. This reduces the safety risks caused by gaze deviation without compromising the efficiency of guidance information transmission.

[0038] To concretize the inventive concept, a typical implementation environment is first provided. The implementation environment involves a vehicle comprising multiple guidance channels for interaction with the driver, such as visual guidance channels, tactile guidance channels, and voice guidance channels; one or more onboard sensor units for acquiring scene parameters (e.g., vehicle state and driver state); and a core onboard operation guide. The onboard operation guide is communicatively connected to the guidance channels and the onboard sensor units, responsible for receiving operation guidance requests from vehicle applications, executing a series of decision-making and scheduling logics, and ultimately controlling the collaborative output of guidance information by each guidance channel. The onboard operation guide typically includes a processor and a memory, the memory storing instructions executable by the processor, which are configured to implement the onboard operation guidance method provided in this application.

[0039] In some embodiments, the on-board sensor unit may include a vehicle speed sensor, an ambient light sensor, and a camera for monitoring the driver's condition. Through these sensors, the on-board operation guide can obtain at least one of the following: vehicle speed information, ambient light intensity information, and driver's line-of-sight and fatigue level information.

[0040] Multiple guidance channels correspond to different sensory modalities, used to present guidance information to the driver in different ways. Therefore, as a possible implementation, guidance channels can include visual guidance channels, tactile guidance channels, and voice guidance channels. Visual guidance channels are implemented using a head-up display (HUD), which projects visual guidance content as a virtual image onto a predetermined display area on the vehicle's windshield. This predetermined display area is located outside the driver's primary field of vision for observing the road ahead, thus avoiding obstruction of the main field of vision. Tactile guidance channels are implemented using vibration units arranged in the steering wheel and driver's seat, which can generate tactile patterns with different vibration frequencies and durations. Voice guidance channels are implemented using speakers in the vehicle's audio system to play voice commands to the driver.

[0041] The in-vehicle operation guide is the core component for implementing guidance control. It can be a processing unit integrated into the in-vehicle infotainment system or a standalone electronic control unit. When an application or function needs to provide operational guidance to the driver, the in-vehicle operation guide receives the guidance request and responds by combining scene parameters obtained from the in-vehicle sensor units. The in-vehicle operation guide can generate guidance content corresponding to the guidance request and allocate this content to one or more guidance channels to drive the corresponding channels to output guidance information. In addition, the vehicle may also include a user interface, such as a central touchscreen, physical buttons on the steering wheel, or a gesture recognition module installed in the vehicle. The user interface is used to collect the driver's operation actions and input commands and send this operation data to the in-vehicle operation guide.

[0042] Figure 1 This is a flowchart illustrating an exemplary embodiment of the vehicle operation guidance method provided in this application, executed by the vehicle operation guide. The following description, in conjunction with... Figure 1 The in-vehicle operation guidance method provided in the embodiments of this application will be described in detail.

[0043] In S110, in response to the operation guidance requirement, the corresponding guidance request is obtained.

[0044] Operation guidance requests refer to any cause or condition that triggers the in-vehicle system to initiate an operation guidance process. This request can encompass various scenarios, including system-initiated identification, user-initiated requests, and application-initiated requests. Specifically, operation guidance requests can originate from, but are not limited to, the following scenarios: First, the system detects that a user is using an application or function for the first time; or the system determines that the user is unfamiliar with the current operation process (e.g., operation time exceeds a preset threshold, repetitive errors occur in the operation sequence, or deviation from the standard operation path). Second, the user explicitly expresses the need for operation guidance through voice commands, touch operations, gestures, or physical buttons. Third, based on the user's operation history, current driving scenario, or safety policy, the system automatically determines that it is necessary to provide guidance to the user in specific environments (such as the first use of a complex function, safety-related operations) to improve operational efficiency or driving safety. Fourth, an application or functional module in the in-vehicle system (such as navigation, driver assistance, and vehicle settings) determines, according to its internal logic, that it needs to guide the user to complete a specific operation (e.g., the navigation system prompts the user to confirm a detour route). When any of the above conditions is met, an operation guidance request is considered to have been generated, and the system generates or obtains the corresponding guidance request accordingly.

[0045] For example, when an application or function in the in-vehicle system detects that the driver needs guidance, such as when the driver is using an application for the first time or when the system determines that the current operation requires assistance, a guidance request will be generated. The in-vehicle operation guide receives the guidance request through the in-vehicle communication bus as a trigger signal to start the subsequent guidance process. The guidance request may contain information about the target application or function that needs guidance, as well as possibly related current operation status data.

[0046] In real-world driving environments with multiple concurrent tasks, there is a risk of conflict and overlap in guidance flows on the same sensory channels when multiple in-vehicle applications may simultaneously initiate guidance requests. To address this, a multi-application guidance conflict coordination mechanism is introduced. Specifically, when multiple applications simultaneously generate guidance requests, a priority attribute is assigned to each corresponding guidance request. This priority attribute includes the application type and / or the urgency of the operation. For example, guidance requests from safety-related applications (such as tire pressure monitoring and collision warning systems) are given high priority, as are requests from emergency operations (such as vehicle troubleshooting). Regular function guidance requests from entertainment systems or navigation applications may be given lower priority. The in-vehicle operation guide maintains a priority arbitration mechanism that allocates resources based on the priority attribute carried by each guidance request. The highest priority guidance request is allowed to occupy the guidance channel first, according to priority order. For other lower-priority guidance requests that cannot temporarily obtain resources, the system manages them in an orderly manner, placing these requests in a temporary waiting queue according to priority or arrival time. Guidance requests with too low a priority and where the user's current focus is not on the request can also be directly rejected to avoid interfering with the driver at inappropriate times. By constructing such a priority-based arbitration and queuing system, the limited sensory guidance channels are treated as a resource that needs to be centrally scheduled. In complex multi-tasking scenarios, guidance information for safety and emergency matters can be delivered to users in a priority manner, ensuring safety during the driving process.

[0047] Furthermore, the priority arbitration mechanism can dynamically schedule and allocate channel resources for concurrent bootstrapping requests based on the following preset conflict resolution strategies: First, an absolute preemptive priority rule: a newly generated bootstrapping request, if its assigned priority attribute is higher than the priority of the currently executing bootstrapping task, is allowed to immediately interrupt the current task and preempt all bootstrapping channel resources. The preempted low-priority bootstrapping task will be suspended, and its current execution state (such as completed steps) will be saved, resuming execution after the high-priority task finishes. Second, a time-division multiplexing rule for concurrent bootstrapping requests with the same priority: for concurrent bootstrapping requests, the system uses time-division multiplexing for serial processing. For example, these requests are sequentially placed into a temporary bootstrapping queue, which follows a first-in-first-out (FIFO) or time-based sorting principle. The system allows only one bootstrapping request at the head of the queue to occupy bootstrapping channel resources at a time. After it finishes execution (or is interrupted by a higher-priority request), the next request is retrieved from the queue for processing. This ensures that bootstrapping tasks of equal importance can obtain channel resources fairly and in order, avoiding information overlap and conflicts. Third, the system employs a waiting queue and timeout rejection mechanism. For guidance requests that cannot temporarily obtain channel resources due to low priority, the system places them in a dedicated waiting queue for management. This queue supports dynamic adjustment; for example, when a high-priority request is completed, the system will retrieve the next highest-priority request from the waiting queue (if the queue supports priority sorting) or in FIFO order. Simultaneously, to prevent the queue from expanding indefinitely or users from waiting too long, the system sets a timeout threshold for each request in the queue. Once a request's waiting time in the queue exceeds the preset threshold (e.g., 30 seconds), the system will automatically determine that the guidance request has timed out and remove it from the queue. It can also optionally send a brief notification to the user to inform them that the guidance request has not been processed.

[0048] In S120, based on the guidance request, at least two different sensory modal guidance channels are determined as target channels from multiple guidance channels.

[0049] A target channel refers to a specific sensory interaction medium or output path that can be selected and used to deliver guidance information to the user. Examples include a voice channel based on in-vehicle speakers, a tactile channel based on steering wheel or seat vibrators, a visual guidance channel based on a head-up display, and a gesture channel based on gesture recognition sensors, or a combination thereof. If the target channel includes a visual guidance channel, the visual guidance channel is configured to visually present the guidance content at a position that does not overlap with the driver's current primary gaze area. The user's current primary gaze area refers to the core area that the driver focuses their gaze on when observing the road ahead, traffic conditions, and road markings in a normal driving posture. Specifically, "not overlapping with the driver's current primary gaze area" means that the pixel distance between the projection position of the visual guidance content on the head-up display and the driver's gaze focus area on the road ahead detected by the driver's gaze tracking system exceeds a preset threshold, thus ensuring that the driver's gaze does not need to leave the road ahead. The visual guidance channel projects guidance content, such as augmented reality (AR) arrows, highlighted outlines, or brief text prompts, onto the display area of ​​the head-up display device. This allows drivers to receive visual guidance information without taking their eyes off the road ahead. They can perceive the guidance information superimposed on the real scene simply through slight eye movements or peripheral vision, thus improving driving safety when reading information.

[0050] In some embodiments, visual guidance channels can be implemented using an in-vehicle HUD (Head-Up Display) or augmented reality (AR) glasses worn by the user. For example, when a user wears AR glasses, visual guidance content generated during the guidance process (such as dynamic arrows, outline highlights, and operation trajectory animations) is directly projected onto the display area of ​​the AR glasses' lenses and superimposed in real time on the real world observed by the user through the lenses. The display area of ​​the AR glasses is configured not to overlap with the user's primary field of vision (i.e., the core field of vision for observing the road ahead through the central area of ​​the AR glasses lenses), for example, it can be located at the lower, side, or upper edge of the lenses. AR glasses can provide a larger field of view (FOV) and more flexible projection positions, further improving the layout flexibility of visual guidance information while maintaining the core safety advantage that the driver's line of sight does not need to deviate from the road ahead.

[0051] The in-vehicle navigation system manages multiple guidance channels, including visual, tactile, and voice guidance channels, each corresponding to a sensory modality. The process of determining the target channel involves selecting a combination from these available channels. This combination must include at least two different sensory modalities; for example, it could include both voice and tactile guidance channels, or visual, voice, and tactile guidance channels simultaneously. This mandatory combination selection ensures that guidance information is not conveyed through a single visual channel, thus preventing drivers from relying on screen reading to obtain guidance information.

[0052] As a possible implementation, visual, tactile, and voice guidance channels can be replaced with brain-computer interface (BCI) guidance channels. BCI guidance channels can generate neural stimulation patterns corresponding to specific guidance commands. For example, through non-invasive (e.g., scalp electrodes) or invasive (e.g., implantable electrodes) BCIs, auditory perception of voice commands, tactile sensation of directionality, or visual perception of AR arrows can be directly simulated in the user's brain, allowing the user to perceive guidance information without relying on external physical channels. Although the hardware interaction method changes, it is still possible to dynamically determine and allocate multiple virtual sensory channels (simulated hearing, simulated touch, and simulated vision) to collaboratively present information based on guidance needs and user status, while avoiding substantial interference with the user's visual attention while driving.

[0053] In some embodiments, to optimize the logic of determining at least two different sensory modal channels from multiple guidance channels, and to rationally select the appropriate target channel from multiple available channels when facing complex application operations, the system can first respond to the guidance request by analyzing the functional structure and operational complexity of the target application to generate multiple task nodes, each corresponding to an operation step. For example, for an application that adjusts air conditioning temperature, it can be decomposed into task nodes such as locating the temperature control area, activating the temperature adjustment mode, and adjusting the temperature value by sliding up or down. By parsing the application's configuration file or based on historical guidance data, the system can identify all the steps required to complete a full functional operation and structure them into a series of task nodes.

[0054] Next, the task dependencies between multiple task nodes are determined; these dependencies represent the execution order of each task node. For example, the task node of activating the temperature regulation mode must be executed after the task node of locating the temperature control area, and adjusting the temperature value depends on the successful completion of activating the temperature regulation mode. This dependency constitutes a directed logical chain guiding the tasks.

[0055] Next, a difficulty coefficient is evaluated for each task node. This coefficient characterizes the complexity of the task node's operation. The evaluation can be based on a combination of factors, including the number of fingers required for the operation, the complexity of the motion trajectory, whether special interactions such as long presses or double-clicks are needed, and the frequency with which the step has caused operational errors in historical data. For example, a simple button click operation might be assigned a low difficulty coefficient, such as level 1, while a complex gesture requiring two-finger rotation and holding might be assigned a higher difficulty coefficient, such as level 4.

[0056] Next, based on the difficulty level of each task node, at least one recommended combination of guidance channels is assigned to each node. The system maintains an internal mapping database that associates different difficulty levels with the most suitable sensory channel combinations. For example, for simple operations with a difficulty level below 2, a combination of voice and touch can be recommended, with a short voice prompt and a directional vibration sufficient for guidance. For complex spatial positioning operations with a difficulty level above 3, a combination of vision, voice, and touch can be recommended, using AR visualizations to overlay arrows on physical controls, supplemented by voice instructions and tactile feedback, to provide the most comprehensive guidance information.

[0057] Finally, based on at least one recommended combination of guidance channels, the target channel is determined. That is, by combining the recommended combinations of guidance channels from all task nodes and taking into account the limitations of the current driving scenario (such as disabling the visual channel when driving at high speed), the target channel to be actually used in the entire guidance process is finally determined.

[0058] This design is precisely planned according to the actual complexity and logical sequence of the operation. By structurally decomposing the task and pre-matching the optimal combination of sensory channels for each step, it ensures that complex steps receive richer and more multi-dimensional guidance support, while simple steps adopt a lightweight guidance configuration, rationally allocate resources, and improve guidance efficiency.

[0059] In S130, multimodal boot content matching the boot request is generated.

[0060] The vehicle-mounted operation guide generates a structured set of guidance content based on the specific application and operation steps targeted by the guidance request. The content set includes guidance sub-content with different formats and presentations for each identified target channel.

[0061] For example, to effectively transform guidance content into an information format that can be precisely executed by various sensory channels, the guidance content can be generated into at least one of the following sub-contents: First, generate voice guidance content, which includes natural language instructions synthesized by text-to-speech technology to instruct the user to perform corresponding operation steps. For example, "Please slide the control knob up to increase the volume." Second, generate haptic guidance content, which includes vibration patterns with directionality or functional relevance. For example, a clockwise vibration pattern can be simulated by sequentially activating vibration units arranged around the steering wheel to guide the user to increase the volume or scroll forward in a list; a single rapid pulse vibration pattern can be used to simulate the tactile sensation of clicking to confirm. Third, generate visual guidance content, which includes at least one of lines, arrows, outline highlighting, or operation trajectory animations to match and superimpose on physical control components inside the vehicle in spatial coordinates, the physical control components being located in the driver's field of vision observed through a non-overlapping display area. These visual elements are defined by precise coordinates in three-dimensional space, strictly matching the position, shape, and size of actual physical control components inside the vehicle (such as a knob, button, or slider). When this visual guidance content is projected through a head-up display, it is observed by the driver through their peripheral vision and superimposed on the image of the actual physical control component on their retina, creating an augmented reality instruction effect like a drawing on a real object. Fourth, gesture guidance content is generated. This includes three-dimensional gesture trajectory animations demonstrating operation methods. These animations can be pre-made or generated in real time, visually demonstrating to the driver the specific actions of a complex operation, such as how to zoom or rotate on a touchpad with two fingers. These guidance sub-contents form a standardized and precisely measurable form of information expression across different sensory modalities, anchoring abstract instructions to the concrete physical world, improving the clarity of the guidance information and the certainty of the user's operation.

[0062] Regarding the timing of guided content triggering, related technologies typically only passively trigger guidance after a user makes an explicit request, resulting in a lag. To improve the reliability of guidance timing, a proactive pre-guidance method based on user intent prediction is introduced. Specifically, an LSTM (Long Short-Term Memory) prediction model is used to continuously analyze the user's current action trends and / or operation history to identify the user's operational intent and determine the target object for the user's next operation. For example, an in-vehicle operation guide can capture the driver's hand movement trajectory and direction through the user interface's touchscreen or gesture recognition module. When the in-vehicle operation guide captures the driver's hand moving towards the volume control knob area on the steering wheel via a camera, and this behavior pattern highly matches the preceding action trend of increasing the volume in the operation history, it predicts that the user's operational intent is to adjust the volume and determines the target object as the volume control knob.

[0063] In response to the identified user's operational intent, the timing and logic of step S140 are adjusted to assign multimodal guidance content to the target channel and initiate output before the user actually touches the target object. For example, 0.5 to 1 second before the driver's finger touches the volume knob, the voice guidance channel begins playing a prompt tone saying "Please rotate the knob to adjust the volume," while the tactile guidance channel prepares a vibration pattern consistent with the rotation direction at the corresponding position on the steering wheel. This processing method advances the system response time from after the user's operation to when the user's intent is generated, allowing the output of multimodal guidance information to seamlessly connect to the driver's operational action flow, eliminating the sense of delay in passive response, providing timely assistance before the driver makes an operational error or hesitates, and making the operating experience smoother.

[0064] In S140, multimodal guidance content is assigned to the target channel to collaboratively present operation guidance information.

[0065] The in-vehicle control system sends various guidance sub-contents to the corresponding target channel hardware. Voice commands are pushed to the speakers of the voice guidance channel for playback, haptic modes are sent to the steering wheel or seat vibration units of the haptic guidance channel for actuation, and AR (Augmented Reality) graphics data is transmitted to the head-up display device of the visual guidance channel for rendering and projection. These channels work collaboratively within the same guidance task time window, according to a preset timing and logical relationship, to synchronously or sequentially deliver guidance information to the driver from multiple sensory dimensions such as hearing, touch, and peripheral vision.

[0066] In some embodiments, to match the in-vehicle operation guidance method with the operational capabilities of different users and solve the problem that a fixed guidance mode cannot adapt to users with different levels of proficiency, this application provides a personalized adjustment scheme based on user proficiency indicators. The adjustment process is independent of a single guidance process, but its results continuously affect the execution of S120 and S130. That is, firstly, historical interaction behavior data associated with the current user is acquired. Historical interaction behavior data refers to historical information that is loaded, recorded, and used to analyze user behavior patterns after successful user authentication. For example, the user's past operation sequences when using various applications, the dwell time at each step, the success or failure status of the operation, historical operation proficiency indicators, and the user's historical feedback on guidance preferences, etc. Historical interaction behavior data is recorded and updated after each user operation and stored in a database associated with the user's identity.

[0067] Then, based on historical interaction data, the current user's operational proficiency index is calculated. The operational proficiency index is a quantifiable value, calculated comprehensively based on three dimensions: operation success rate, the ratio of operation completion time to target time, and error rate. The operation success rate is the ratio of the number of times a user successfully completes an operation to the total number of attempts, reflecting the accuracy of the operation. The ratio of operation completion time to target time reflects the user's operational efficiency; the closer the ratio is to or less than 1, the more proficient the user is. The error rate refers to the proportion of incorrect operations during the process. By assigning different weights to these three dimensions and performing a weighted sum, a comprehensive index that objectively reflects the user's mastery of a specific application or function—the operational proficiency index—is obtained.

[0068] Finally, based on the user's proficiency index, the level of detail in the multimodal guidance content or the combination of guidance channels is adjusted when executing S120 and S130. The adjustment strategy is as follows: the higher the user's proficiency index, the simpler the guidance; the lower the user's proficiency index, the more detailed the guidance. By quantifying the user's operation history into a continuous and comparable value, the system can dynamically distinguish between novice and expert users, thereby providing differentiated guidance strategies. This avoids unnecessary disturbance to experienced users while providing more comprehensive support to less experienced users.

[0069] As one possible implementation, the operational proficiency index is calculated using the following weighted summation formula: Proficiency=(SuccessfulOperations / TotalOperations)×0.6+(CompletionTimeTarget / ActualCompletionTime)×0.3+(1-ErrorRate)×0.1 In the weighted summation formula, Proficiency represents user proficiency, SuccessfulOperations represents the number of successful operations, TotalOperations represents the total number of operations, CompletionTimeTarget represents the target completion time, ActualCompletionTime represents the actual completion time, and ErrorRate represents the error rate. SuccessfulOperations / TotalOperations represents the success rate, with a weight of 0.6, used to measure the accuracy of user operations and reflect the user's mastery of the operation process. CompletionTimeTarget / ActualCompletionTime represents the efficiency ratio, with a weight of 0.3, comparing the user's actual completion time with the system's preset target time to reflect the user's proficiency and fluency in performing the operation. The closer this ratio is to or greater than 1, the faster and closer the user's operation is to the ideal state. (1-ErrorRate) represents the error-free rate, with a weight of 0.1, used to measure the error rate during the operation process, serving as a supplementary correction to the accuracy and efficiency of the operation. The final calculated Proficiency value is a continuous value between 0 and 1. The higher the value, the higher the user's proficiency in operating a specific application or function.

[0070] To provide an executable decision-making logic for proficiency-based adjustments, as one possible implementation, the in-vehicle operation guide internally presets a first threshold and a second threshold, with the first threshold being greater than the second. These two thresholds divide the proficiency index range of 0 to 1 into at least three intervals. When the operation proficiency index is higher than the first threshold, indicating that the user is a highly proficient user, the system will execute a simplification strategy, simplifying the multimodal guidance content and / or reducing the number of target channels. For example, detailed step-by-step explanations may be reduced to keyword prompts, or the number of target channels may be reduced while ensuring safety; for extremely simple operations, only a brief haptic feedback may be triggered, without playing voice prompts. When the operation proficiency index is lower than the second threshold, indicating that the user is a novice user, the system will execute an enhancement strategy, increasing the multimodal guidance content, increasing the number of guidance repetitions, or enabling more guidance channels. For example, increasing the detail of the multimodal guidance content, providing more detailed step-by-step explanations, increasing the number of guidance repetitions to ensure the user hears and understands, or enabling more guidance channels, adding AR visual path guidance to an operation that was originally only guided by voice and haptics to provide maximum assistance. By setting clear high and low thresholds, a deterministic decision boundary is provided for the system to dynamically adjust the guidance intensity according to the user's capabilities, ensuring the stability and response speed of the adjustment logic.

[0071] To achieve continuous evolution and personalized adaptation of guidance strategies, some embodiments may also provide a reinforcement learning-based guidance effectiveness evaluation and feedback optimization mechanism. This mechanism is triggered after each guidance task is completed, quantifying the guidance effect and using it as a feedback signal to drive iterative optimization of the guidance strategy decision-making logic. For example, the guidance effectiveness evaluation process consists of the following multi-dimensional indicators: operation success rate, which refers to the proportion of users who successfully complete the operations instructed in this guidance. This indicator measures the accuracy and effectiveness of the guidance information from the outcome perspective; completion time, which refers to the time elapsed from the start of the multimodal guidance content output to the user completing the final operation. This indicator reflects the smoothness of the guidance process and the user's comprehension speed from an efficiency perspective; and user satisfaction, which can be achieved by triggering a brief voice or button interaction immediately after the guidance ends to collect the user's subjective feedback rating. This indicator directly reflects the user's acceptance of the guidance method, content, and intensity from an experience perspective. After collecting the above multi-dimensional indicators, the system uses the quantified evaluation results as a reward signal and inputs them into a reinforcement learning model. The model's state space includes at least the user's operational proficiency index, current scene parameters, and the complexity of the guidance task; the action space includes strategies such as adjusting the detail of multimodal guidance content, adding, removing, or switching guidance channel combinations, and adjusting the number of guidance repetitions.

[0072] Reinforcement learning models aim to maximize long-term cumulative rewards. They dynamically update a decision network by learning the optimal mapping strategy from state to action. The adjustment logic can include: for users with high operational proficiency, when the success rate and efficiency metrics remain consistently high, the system automatically tends to simplify guidance content, such as reducing the length of voice commands, decreasing the detail of AR visualization elements, or retaining only key haptic cues. For error-prone steps, i.e., steps with low success rates in multiple guidance sessions, the system increases the guidance details and repetition frequency for that step, such as playing more detailed voice breakdowns, overlaying more persistent dynamic trajectory animations on the AR-HUD, or providing haptic confirmation signals before and after the operation. Based on user satisfaction feedback, the system can adjust the frequency and combination of different guidance channels. For example, if a user's satisfaction rating for pure voice guidance is consistently higher than that for the combination of voice and AR, the model will gradually reduce the weight of the visual channel in subsequent guidance tasks for that user.

[0073] Through multiple rounds of guided interaction and reinforcement learning iterations, the model can adaptively converge to a guidance strategy that is highly fitted to the specific user's operational capabilities and preferences, thereby continuously improving the matching degree of guidance content and user satisfaction, thus realizing the self-learning and personalized evolution capabilities of the guidance system.

[0074] In order to enable the guidance behavior to adapt to the dynamically changing driving scenarios and solve the problem that the fixed guidance configuration may fail or even cause danger in specific scenarios, this application also provides a dynamic channel reconfiguration mechanism based on real-time scene perception. The reconfiguration mechanism is a parallel monitoring and intervention process that continuously and dynamically corrects the target channel determined in step S120 and the presentation process in step S140.

[0075] The in-vehicle operation guide continuously acquires scene parameters characterizing the current driving scenario through in-vehicle sensor units. These scene parameters can be obtained from vehicle speed sensors, ambient light sensors, and cameras, including at least one of vehicle speed, ambient light intensity, and driver fatigue status. Vehicle speed information reflects the vehicle's motion state; high-speed driving usually means higher driving load and risk. Ambient light intensity affects the readability of visual display devices; strong light may obscure the displayed content, while excessively bright displays in low light can be glaring. Driver fatigue status is determined by analyzing physiological parameters such as the driver's blink frequency, changes in steering wheel grip force, or heart rate; when fatigued, the driver's reaction time and information processing ability decline.

[0076] The in-vehicle operation guide internally stores a set of preset scene guidance strategy mapping rules. These rules define the correspondence between different scene parameters or combinations thereof and the operating status of the guidance channels. Based on these mapping rules and real-time acquired scene parameters, the operating parameters of some or all guidance channels are enabled, disabled, or adjusted in real time. For example, when the vehicle speed exceeds a preset safety threshold, the preset rules trigger an instruction to disable the visual guidance channels to avoid any unnecessary visual information distracting the driver during critical moments of high-speed driving. When the ambient light intensity exceeds a preset comfortable reading range, the rules trigger adjustments to the brightness and contrast parameters of the visual guidance channel display content to ensure that AR graphics remain clearly visible and do not cause glare in excessively strong or weak light. When driver fatigue is detected, the rules trigger instructions to simplify the multimodal guidance content structure, using shorter, more direct voice commands and stronger tactile stimulation to deliver guidance information, reducing the user's cognitive load. By establishing a direct mapping relationship between the physical parameters of the driving scenario and the working status of the guidance channel, the guidance system can switch the guidance strategy to the most suitable modal combination and parameter configuration for the current working condition in real time, just like having environmental perception capabilities, thereby ensuring the effectiveness and safety of the guidance service in various environments.

[0077] Based on the established mapping between scene parameters and guidance strategies, a robust safety control logic can be developed to address the issue of visual information interfering with drivers in high-risk scenarios, particularly regarding the visual guidance channel—a key safety risk point. For example, the first measure is that when the vehicle speed exceeds 60 km / h, the onboard operator will unconditionally disable the visual guidance channel, immediately stopping all AR graphics and text projection on the head-up display, retaining only non-visual channels such as voice and touch for guidance. The second measure is that when the ambient light intensity detected by the ambient light sensor exceeds the preset comfortable display range, the onboard operator will dynamically adjust the brightness and contrast parameters of the visual guidance channel display to adapt to changes in ambient light. For example, in low-light environments such as at night or in tunnels, the display brightness is automatically reduced to the minimum visible level to avoid stimulating the eyes under dark adaptation; in strong sunlight, the brightness and contrast are maximized to ensure that graphic elements can penetrate the strong light and be clearly perceived by the driver in their surroundings. The third measure is that when camera or other devices detect driver fatigue, the in-vehicle guidance system simplifies the structure of the entire multimodal guidance content. This includes streamlining visual guidance content, such as stopping complex trajectory animations and retaining only the core static highlight indicators of target locations, to reduce the cognitive burden on fatigued drivers. By programmatically binding the three key safety factors—vehicle speed, lighting conditions, and driver status—to the on / off state and parameters of the visual channel, it ensures that the guidance system itself does not become a source of distraction for the driver in high-risk scenarios.

[0078] Below, a typical use case is shown in a vehicle that can use the in-vehicle operation guidance method provided in this application: The scenario is set as a driver starting the vehicle and trying for the first time to use a newly upgraded application with a certain degree of operational complexity in the in-vehicle system.

[0079] The first step is to start the vehicle system and begin the boot process.

[0080] The second step involves the system performing rapid facial recognition via the in-vehicle camera or logging in using the account information provided by the driver to complete user identity authentication and load the corresponding personalized configuration.

[0081] The third step involves the system loading the user's historical behavior data and preference settings, including past operating habits when using various applications, common error patterns, and calculated operation proficiency indicators.

[0082] The fourth step is to initialize all available multimodal guidance channels, including voice channels, haptic channels (implemented through steering wheel or seat vibration units), AR visualization channels (implemented through AR-HUD), gesture channels, and facial expression channels, and configure the initial operating parameters of each channel.

[0083] Fifth, the driver clicked the newly added application icon on the central control screen, and the system detected that this was the first time the user had used the application.

[0084] The sixth step involves the system analyzing the user's historical behavior data (e.g., the user has a low success rate with complex operations) to determine if the application needs to be guided through the startup process.

[0085] The seventh step involves the system analyzing the application's functional structure, breaking down a typical operation process into multiple task nodes, determining the dependencies between nodes and their respective difficulty levels, and generating a guided task tree.

[0086] The eighth step involves the system generating multimodal guidance content based on the guidance task tree and user attributes, including voice commands, tactile vibration patterns, and AR visualization elements (such as dynamic arrows and outline highlighting).

[0087] The ninth step involves the system monitoring the driving scene in real time using onboard sensors to obtain scene parameters such as vehicle speed, ambient light intensity, and driver status.

[0088] The tenth step involves the system dynamically adjusting its guidance strategy based on scene perception results. For example, when the vehicle speed is below a preset safety threshold, the AR-HUD visual channel remains open; when the vehicle speed exceeds the threshold, the visual channel is automatically disabled, retaining only voice and tactile guidance.

[0089] In the eleventh step, the system, taking into account user attributes (such as proficiency) and the current scenario, decides to use a triple collaborative guidance combination of voice, touch, and AR as the target channel combination for this guidance.

[0090] The twelfth step involves the system using a built-in prediction model (such as LSTM) to analyze the user's current hand movement trajectory and recent operation history to predict the target object that the user will perform next.

[0091] Step 13: The system determines the outcome based on the prediction results: if the user's intention is successfully predicted, proactive pre-guidance will be initiated 0.5-1 seconds before the target object is actually touched; if not predicted, guidance will be provided as needed when the user initiates an operation request.

[0092] Step fourteen: The system performs multimodal guidance, presenting operation guidance information to the user through the selected target channel, including voice prompts, tactile vibration, and AR visual guidance.

[0093] Step 15: After the guidance is completed, the system will quantitatively evaluate the effectiveness of the guidance based on indicators such as operation success rate, completion time, and user satisfaction rating.

[0094] Step sixteen: The system determines whether the user profile needs to be updated based on the evaluation results. If an update is needed, proceed to the next step; otherwise, skip to step eighteen.

[0095] In the seventeenth step, the system uses the evaluation results as a reward signal and inputs them into the reinforcement learning model to continuously optimize the guidance strategy decision-making logic, while updating the user's historical behavior data and proficiency indicators.

[0096] Step 18: The system checks for conflicts where multiple applications need to be booted simultaneously.

[0097] Step 19: If a boot conflict exists, the system resolves the conflict according to a priority arbitration mechanism: the highest priority boot request is allowed to immediately occupy channel resources, while other requests are placed in a waiting queue in order or rejected directly. For example, a high-priority security alarm request can immediately interrupt the current low-priority application boot process.

[0098] Step 20: The guidance process is complete, and the driver can use the application normally. Throughout the process, the driver's eyes never leave the road ahead for an extended period, achieving a safe and efficient guidance experience.

[0099] Understandably, the conflict arbitration mechanism in steps nineteen and twenty is an independent, parallel service—a self-contained module that resides in memory and continuously monitors. At any moment, if a new boot request is detected with a higher priority than the currently executing request, arbitration and preemption are immediately triggered.

[0100] Figure 2 This is a schematic diagram of the structure of an on-board operation guidance device provided in an exemplary embodiment of this application, as shown below. Figure 2 As shown, the vehicle-mounted operation guidance device 2 includes: an acquisition module 21, used to acquire a corresponding guidance request in response to operation guidance needs; a channel determination module 22, used to determine at least two different sensory modal guidance channels as target channels from multiple guidance channels based on the guidance request; wherein, if the target channel includes a visual guidance channel, the visual guidance channel is configured to: visually present the guidance content at a position that does not overlap with the driver's current main field of vision; a content generation module 23, used to generate multimodal guidance content that matches the guidance request; and a guidance execution module 24, used to allocate the multimodal guidance content to the target channels to collaboratively present operation guidance information.

[0101] As one possible implementation, the channel determination module 22 can be configured to: respond to a bootstrapping request, analyze the functional structure and operational complexity of the target application to generate multiple task nodes, each task node corresponding to an operation step; determine the task dependencies between the multiple task nodes; wherein the task dependencies characterize the execution order between the task nodes; evaluate the difficulty coefficient for each task node; wherein the difficulty coefficient characterizes the operational complexity of the task node; based on the difficulty coefficient of each task node, assign at least one recommended bootstrapping channel combination to each task node; and determine the target channel based on at least one recommended bootstrapping channel combination.

[0102] As one possible implementation, the channel determination module 22 can be configured to: analyze the user's current action trends and / or operation history, identify the user's operation intention, and determine the target operation object of the user's next step; wherein, the multimodal guidance content is allocated to the target channel to collaboratively present operation guidance information, including: in response to the user's operation intention, allocating the multimodal guidance content to the target channel and starting output before the user actually contacts the target operation object.

[0103] As one possible implementation, the channel determination module 22 can also be configured to: acquire historical interaction behavior data associated with the current user; calculate the current user's operation proficiency index based on the historical interaction behavior data; wherein the operation proficiency index is calculated based on the operation success rate, the ratio of operation completion time to target time, and the error rate; and adjust the level of detail of the multimodal guidance content or the combination of guidance channels based on the operation proficiency index.

[0104] As one possible implementation, the channel determination module 22 can also be configured to: simplify the multimodal guidance content and / or reduce the number of target channels when the operation proficiency index is higher than the first threshold; and increase the multimodal guidance content, increase the number of guidance repetitions, or enable more guidance channels when the operation proficiency index is lower than the second threshold; wherein the first threshold is greater than the second threshold.

[0105] As one possible implementation, the channel determination module 22 can also be configured to: acquire scene parameters representing the current driving scenario; wherein the scene parameters include at least one of vehicle speed, ambient light intensity and driver fatigue state; and enable, disable or adjust the operating parameters of the guidance channel in real time based on preset scene guidance strategy mapping rules and scene parameters.

[0106] As one possible implementation, the channel determination module 22 can also be configured to: disable the visual guidance channel when the vehicle speed is greater than or equal to a preset safety threshold; adjust the brightness and contrast parameters of the visual guidance channel display content when the ambient light intensity exceeds a preset range; and simplify the structure of the multimodal guidance content when driver fatigue is detected.

[0107] As one possible implementation, the acquisition module 21 can be configured to: when multiple applications generate operation guidance requirements at the same time, assign a priority attribute to each corresponding guidance request; wherein, the priority attribute includes the application type and / or the urgency of the operation; based on a preset priority arbitration mechanism, allow the highest priority guidance request to occupy the guidance channel in order of priority, and put other guidance requests into the waiting queue in order or reject them directly.

[0108] As one possible implementation, the content generation module 23 can also be configured to generate at least one of the following guidance sub-contents: generating voice guidance content; wherein the voice guidance content includes natural language instructions for instructing the user to perform corresponding operation steps; generating haptic guidance content; wherein the haptic guidance content includes vibration patterns with directionality or functional relevance; generating visual guidance content; wherein the visual guidance content includes at least one of lines, arrows, outline highlighting, or operation trajectory animation, for matching with physical control components inside the vehicle in spatial coordinates and superimposing them on the physical control components, which are located in the driver's field of vision observed through a non-overlapping display area; generating gesture guidance content; wherein the gesture guidance content includes a three-dimensional gesture trajectory animation for demonstrating the operation method.

[0109] An electronic device includes: a processor; a memory for storing processor-executable instructions; and a processor for executing the vehicle operation guidance method described in the embodiments of this application.

[0110] Below, for reference Figure 3 This application describes an electronic device according to embodiments thereof. The electronic device may be either or both of a first device and a second device, or a standalone device independent of them, which may communicate with the first device and the second device to receive acquired input signals from them.

[0111] Figure 3 A block diagram of an electronic device according to an embodiment of this application is illustrated.

[0112] like Figure 3 As shown, the electronic device 30 includes one or more processors 31 and memory 32.

[0113] The processor 31 may be a central processing unit (CPU) or other form of processing unit with data processing and / or instruction execution capabilities, and may control other components in the electronic device 30 to perform desired functions.

[0114] The memory 32 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 31 may execute the program instructions to implement the vehicle operation guidance methods of the various embodiments of this application described above and / or other desired functions. Various contents such as input signals, signal components, and noise components may also be stored in the computer-readable storage medium.

[0115] In one example, the electronic device 30 may also include an input device 33 and an output device 34, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0116] When the electronic device is a standalone device, the input device 33 can be a communication network connector for receiving the collected input signals from the first device and the second device.

[0117] In addition, the input device 33 may also include, for example, a keyboard, a mouse, etc.

[0118] The output device 34 can output various information to the outside, including determined distance information, direction information, etc. The output device 34 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0119] Of course, for the sake of simplicity, Figure 3 Only some of the components of the electronic device 30 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 30 may include any other suitable components depending on the specific application.

[0120] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0121] A computer-readable storage medium stores a computer program for executing the vehicle operation guidance method described in the embodiments provided in this application.

[0122] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0123] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A vehicle-mounted operation guidance method, characterized in that, include: In response to the operation guidance requirements, obtain the corresponding guidance request; Based on the guidance request, at least two guidance channels with different sensory modalities are determined from multiple guidance channels as target channels; wherein, if the target channel includes a visual guidance channel, the visual guidance channel is configured to: visually present the guidance content at a position that does not overlap with the driver's current main field of vision. Generate multimodal boot content that matches the boot request; The multimodal guidance content is allocated to the target channel to collaboratively present operation guidance information.

2. The vehicle-mounted operation guidance method according to claim 1, characterized in that, The step of determining at least two different sensory modalities as target channels from multiple guidance channels based on the guidance request includes: In response to the guidance request, the functional structure and operational complexity of the target application are analyzed to generate multiple task nodes, each of which corresponds to an operation step; Determine the task dependencies between the multiple task nodes; wherein, the task dependencies represent the execution order of each task node; A difficulty coefficient is evaluated for each task node; wherein the difficulty coefficient is used to characterize the operational complexity of the task node; Based on the difficulty coefficient of each task node, at least one recommended combination of guidance channels is assigned to each task node; The target channel is determined based on at least one recommended combination of guiding channels.

3. The vehicle-mounted operation guidance method according to claim 1, characterized in that, Before allocating the multimodal guidance content to the target channel to collaboratively present the operation guidance information, the method further includes: Analyze the user's current action trends and / or operation history to identify the user's operation intentions and determine the target of the user's next operation. The multimodal guidance content is allocated to the target channel to collaboratively present operation guidance information, including: In response to the user's operational intent, the multimodal guidance content is allocated to the target channel and output is initiated before the user actually interacts with the target operation object.

4. The vehicle-mounted operation guidance method according to claim 1, characterized in that, The method further includes: Retrieve historical interaction data associated with the current user; Based on the historical interaction data, the current user's operation proficiency index is calculated; wherein, the operation proficiency index is calculated based on the operation success rate, the ratio of operation completion time to target time, and the error rate; Based on the operational proficiency index, adjust the level of detail of the multimodal guidance content or the combination of guidance channels.

5. The vehicle-mounted operation guidance method according to claim 4, characterized in that, The adjustment of the level of detail of the multimodal guidance content or the combination of guidance channels based on the operational proficiency index includes: When the operational proficiency index is higher than the first threshold, simplify the multimodal guidance content and / or reduce the number of target channels; When the operation proficiency index is lower than the second threshold, increase the multimodal guidance content, increase the number of guidance repetitions, or enable more guidance channels; Wherein, the first threshold is greater than the second threshold.

6. The vehicle-mounted operation guidance method according to claim 1, characterized in that, The method further includes: Obtain scene parameters that characterize the current driving scenario; wherein, the scene parameters include at least one of vehicle speed, ambient light intensity, and driver fatigue state; Based on the preset scene guidance strategy mapping rules and the scene parameters, the operating parameters of the guidance channel are enabled, disabled, or adjusted in real time.

7. The vehicle-mounted operation guidance method according to claim 6, characterized in that, The method of enabling, disabling, or adjusting the operating parameters of the guidance channel in real time based on the preset scene guidance strategy mapping rules and the scene parameters includes: When the vehicle speed is greater than or equal to a preset safety threshold, the visual guidance channel is disabled. When the ambient light intensity exceeds the preset range, adjust the brightness and contrast parameters of the content displayed in the visual guidance channel; When driver fatigue is detected, the structure of the multimodal guidance content is simplified.

8. The vehicle-mounted operation guidance method according to claim 1, characterized in that, The step of responding to the operation guidance requirement and obtaining the corresponding guidance request includes: When multiple applications simultaneously generate operation guidance requests, a priority attribute is assigned to each corresponding guidance request; wherein, the priority attribute includes application type and / or operation urgency. Based on a preset priority arbitration mechanism, the highest priority boot request is allowed to occupy the boot channel according to priority order, while other boot requests are placed in the waiting queue or rejected directly.

9. The vehicle-mounted operation guidance method according to claim 1, characterized in that, The generation of multimodal boot content matching the boot request also includes: Generate at least one of the following bootstrap contents: Generate voice guidance content; wherein, the voice guidance content includes natural language instructions to instruct the user to perform corresponding operation steps; Generate tactile guidance content; wherein, the tactile guidance content includes vibration patterns with directionality or functional relevance; Generate visual guidance content; wherein the visual guidance content includes at least one of lines, arrows, outline highlights or operation trajectory animations, used to match the physical control components inside the vehicle in spatial coordinates and superimpose them on the physical control components, which are located in the field of vision observed by the driver through the non-overlapping display area; Generate gesture guidance content; wherein, the gesture guidance content includes a three-dimensional gesture trajectory animation for demonstrating the operation method.

10. A vehicle, characterized in that, include: The guidance channel includes a visual guidance channel, a tactile guidance channel, and a voice guidance channel; The vehicle-mounted sensor unit is used to collect scene parameters; An on-board operation guide, which is communicatively connected to the guidance channel and the on-board sensor unit, is used to perform the on-board operation guidance method according to any one of claims 1-9.

Images