Display method and device of indication information, vehicle and computer readable storage medium

Abstract

The invention provides an indication information display method and device, a vehicle and a computer readable storage medium, and belongs to the technical field of intelligent cabins, and the method comprises the steps: obtaining multi-mode driving monitoring data, and predicting the driving intention of a driver based on the multi-mode driving monitoring data; obtaining the influence degree of a sensing object in the surrounding environment of the vehicle on the driving task of the vehicle, and carrying out weighted calculation on the influence degree of the sensing object on the driving task of the vehicle to obtain the saliency of the sensing object; and displaying the indication information corresponding to at least part of the perception objects on the front windshield in an AR form based on the saliency of the perception objects. According to the method, the indication information can be rendered based on the predicted driving intention of the driver and the current driving situation, the information most needed by the driver is provided, and then potential safety hazards caused by visual interference to the driver due to information overload can be avoided.

Description

Technical Field

[0001] This application relates to the field of smart cockpit technology, and in particular to a method, device, vehicle, and computer-readable storage medium for displaying instruction information. Background Technology

[0002] Augmented Reality Head-Up Display (AR-HUD) technology, as the core of the next generation of in-vehicle human-machine interaction, aims to integrate virtual information with the real driving environment to improve driving safety and experience.

[0003] However, current AR-HUD systems on the market suffer from information overload and piling up. This "information piling up" can cause visual interference for drivers, distract their attention, increase cognitive load, and ultimately create safety hazards. Furthermore, because existing AR-HUD technology lacks both driving context awareness and the ability to predict driver intentions, the displayed content is typically rendered based on fixed content and rules, failing to display content that matches the driver's intentions and the driving context, and thus unable to provide on-demand display. Summary of the Invention

[0004] In view of the above problems, this application provides a method, apparatus, vehicle, and computer-readable storage medium for displaying indication information that overcomes or at least partially solves the above problems. The technical solution is as follows: A method for displaying instruction information, applied to a vehicle, the method comprising: acquiring multimodal driving monitoring data and predicting the driver's driving intention based on the multimodal driving monitoring data, wherein the multimodal monitoring data includes one of the following: driver monitoring data, vehicle control signals, and navigation path information; acquiring the influence degree of perceived objects in the vehicle's surrounding environment on the vehicle's driving task, wherein the influence degree of the perceived objects on the vehicle's driving task includes: the correlation degree with the driver's driving intention, and the influence degree of the perceived objects on the vehicle's driving task further includes at least one of the following: the threat level based on the motion trajectory and the correlation degree with the navigation path information; performing a weighted calculation on the influence degree of the perceived objects on the vehicle's driving task to obtain the salience of the perceived objects; and displaying at least a portion of the instruction information corresponding to the perceived objects in AR form on the windshield based on the salience of the perceived objects.

[0005] Optionally, the driver monitoring data includes one of the following: the driver's head posture, hand movements, body posture, and eye gaze point; the vehicle control signals include one of the following: steering wheel angle, turn signal status, and input mode of acceleration / deceleration pedals; predicting the driver's driving intention based on the multimodal driving monitoring data includes: inputting the multimodal driving monitoring data into a pre-trained intention prediction model, obtaining the probability distribution of various driving intentions output by the intention prediction model, and selecting the driving intention with the highest probability value as the driver's driving intention.

[0006] Optionally, obtaining the influence of perceived objects in the vehicle's surrounding environment on the vehicle's driving task includes: inputting the driver's driving intention and the intention features of the perceived objects into a pre-trained correlation network model, and using the output of the correlation network model as the correlation degree with the driver's driving intention; if the influence of the perceived objects on the vehicle's driving task includes the threat level based on the motion trajectory, inputting the vehicle's driving trajectory features and the motion trajectory features of the perceived objects into a pre-trained correlation network model, and using the output of the correlation network model as the threat level based on the motion trajectory; if the influence of the perceived objects on the vehicle's driving task includes the correlation degree with the navigation path information, inputting the vehicle's navigation path information, the identification information of the perceived objects, and their location information into a pre-trained correlation network model, and using the output of the correlation network model as the correlation degree with the navigation path information.

[0007] Optionally, the weighted calculation of the impact of the perceived object on the driving task of the vehicle to obtain the saliency of the perceived object includes: normalizing each parameter of the impact of the driving task, and then multiplying it by the weight corresponding to each parameter to obtain the saliency of the perceived object, wherein the weight corresponding to each parameter is adjusted according to the vehicle state and driving scenario of the vehicle.

[0008] Optionally, displaying at least a portion of the indication information corresponding to the perceived object in AR form on the windshield based on the salience of the perceived object includes: displaying at least a portion of the indication information corresponding to the perceived object in AR form on the windshield when the salience of the perceived object is greater than the rendering threshold of the indication information corresponding to the at least a portion of the perceived object, wherein the indication information corresponding to the perceived object includes at least one of the following: safety warning indication information, navigation guidance indication information, driving assistance indication information, and ecological information indication information, wherein the rendering threshold of the safety warning indication information > the rendering threshold of the navigation guidance indication information > the rendering threshold of the driving assistance indication information > the rendering threshold of the ecological information indication information.

[0009] Optionally, the sensing objects include multiple objects; the step of displaying at least some of the sensing objects' corresponding indication information in AR form on the windshield includes: sorting the salience of the sensing objects, and according to the order of salience from high to low, decreasing the rendering display degree of the indication information corresponding to the sensing objects in sequence, and displaying at least some of the sensing objects' corresponding indication information in AR form on the windshield.

[0010] Optionally, displaying at least a portion of the indication information corresponding to the perceived object in AR form on the windshield includes: enhancing the display of the safety warning indication information when at least a portion of the indication information corresponding to the perceived object includes safety warning indication information and the rendering threshold of the safety warning indication information is the threshold of the highest safety level, and suppressing the display of other indication information in at least a portion of the indication information corresponding to the perceived object.

[0011] A display device for indicating information includes: a driving intention prediction module, configured to acquire multimodal driving monitoring data and predict the driver's driving intention based on the multimodal driving monitoring data, wherein the multimodal driving monitoring data includes one of the following: driver monitoring data, vehicle control signals, and navigation path information; a perception object analysis module, configured to calculate the influence degree of perception objects in the vehicle's surrounding environment on the vehicle's driving task, and to perform a weighted calculation on the influence degree of the perception objects on the vehicle's driving task to obtain the salience of the perception objects; wherein the influence degree of the perception objects on the vehicle's driving task includes: the correlation degree with the driver's driving intention, and the influence degree of the perception objects on the vehicle's driving task also includes at least one of the following: the threat level based on the movement trajectory and the correlation degree with the navigation path information; and a display module, configured to display at least a portion of the indication information corresponding to the perception objects in AR form on the windshield based on the salience of the perception objects.

[0012] A vehicle including at least one processor, which executes steps of the method for displaying indication information as described above.

[0013] A computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the method for displaying indication information as described in any of the preceding claims.

[0014] By means of the above technical solution, this application provides a method, device, vehicle and computer-readable storage medium for displaying instruction information, which is applied to a vehicle. It can render and display instruction information based on the predicted driving intention of the driver and the current driving situation, realize intelligent AR rendering, display content that matches the driving intention and the current driving situation, provide the driver with the most needed information, and thus avoid the safety hazards caused by visual interference to the driver due to information overload.

[0015] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, specific embodiments of this application are given below. Attached Figure Description

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 A schematic flowchart of a method for displaying indication information provided in an embodiment of this application is shown; Figure 2 This illustration shows a schematic flowchart of a method for displaying indication information based on the salience of a perceived object, according to an embodiment of this application. Figure 3 A schematic flowchart of a method for displaying indication information provided in an embodiment of this application is shown; Figure 4 This illustration shows a schematic diagram of the structure of a display device for indicating information provided in an embodiment of this application; Figure 5 A schematic diagram of the structure of a vehicle provided in an embodiment of this application is shown. Detailed Implementation

[0017] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0018] AR-HUD technology, as the core of next-generation in-vehicle human-machine interaction, aims to integrate virtual information with the real driving environment to improve driving safety and experience. However, current AR-HUD systems on the market, whether pre-installed or aftermarket, generally suffer from the following bottlenecks: 1. Information Overload and Visual Interference: Existing AR-HUD systems tend to overwhelm the driver's field of vision with as much information as possible (such as vehicle speed, RPM, navigation, media, and Advanced Driver Assistance System (ADAS) warnings) without prioritizing any particular information. In scenarios requiring high concentration, such as complex urban intersections or high-speed driving, this "information overload" can severely distract the driver, increase cognitive load, and ultimately create safety hazards.

[0019] 2. Lack of Context Awareness and Information Prioritization: Existing AR-HUD systems typically render content based on fixed, preset rules, lacking a deep understanding of the current driving context. Current systems cannot intelligently determine which information is most important to the driver at any given moment, nor can they dynamically adjust information priority based on the driver's intended next action. For example, regardless of whether the driver is preparing to change lanes, the intensity and method of blind spot warnings remain unchanged, failing to provide "on-demand reminders."

[0020] 3. Passive interaction and weak decision support capabilities: Existing AR-HUD systems are still passive information projection devices, unidirectionally "feeding" information to the driver. They cannot predict the driver's intentions and therefore cannot proactively and forward-lookingly provide decision support.

[0021] In summary, the core logic of current AR-HUD systems is to "display as much information as possible," rather than "displaying the most needed information at the most appropriate time." Information overload can cause visual interference for drivers, leading to safety hazards. Furthermore, because current AR-HUD technology lacks both driving context awareness and the ability to predict driver intentions, the displayed content is usually rendered based on fixed content and rules, failing to display content that matches the driver's intentions and the driving context, and thus failing to achieve on-demand display.

[0022] To address the problems existing in the prior art, embodiments of this application provide a method for displaying indication information, such as... Figure 1 As shown, Figure 1 This is a schematic flowchart illustrating a method for displaying indication information provided in an embodiment of this application. The method is applicable to vehicles and includes the following steps (S101-S104): S101. Acquire multimodal driving monitoring data and predict the driver's driving intention based on the multimodal driving monitoring data; In some embodiments, vehicle multimodal monitoring data includes one of the following: (1) driver monitoring data, (2) vehicle control signals, and (3) navigation path information.

[0023] The following section provides a detailed explanation and illustration of vehicle multimodal monitoring data, using specific examples.

[0024] (1) Driver monitoring data: This refers to some posture data of the driver related to driving intentions, acquired through the camera of the Driver Monitoring System (DMS). In some embodiments, driver monitoring data includes one of the following: the driver's head posture, hand movements, body posture, and eye gaze point. For example, the driver's head turning angle and speed; if the driver's head turns to the left, they may be looking at the left rearview mirror, indicating that the driver may want to turn left or change lanes to the left. Another example is the eye gaze point observing the left / right rearview mirror, the central control screen, the instrument panel, etc. It is understood that the above-listed driver monitoring data are merely illustrative examples, and the embodiments of this application do not limit the driver monitoring data.

[0025] (2) Vehicle control signals: These are vehicle driving status data acquired via the CAN bus. In some embodiments, vehicle control signals include one of the following: steering wheel angle, turn signal status, and input mode of the acceleration / deceleration pedals. For example, when the driver turns the steering wheel to the left, the vehicle control signal indicates that the steering wheel should turn to the left. As another example, when the driver activates the right turn signal, the vehicle control signal indicates that the right turn signal should flash. It is understood that the above-listed vehicle control signals are merely illustrative examples, and the embodiments of this application do not limit the vehicle control signals.

[0026] (3) Navigation route information: This refers to the upcoming maneuver, route, and prompts obtained from the vehicle navigation system. For example, "Turn right at the intersection ahead" or "Enter the highway ramp." It is understood that the above-listed navigation route information is only illustrative and this application does not limit the navigation route information.

[0027] In this embodiment, the driver's driving intention can be predicted using the aforementioned vehicle multimodal monitoring data, so that the subsequent rendering of instruction information can be based on the driver's driving intention.

[0028] In some embodiments, predicting a driver's driving intention based on multimodal driving monitoring data includes: inputting the multimodal driving monitoring data into a pre-trained prediction model, obtaining the probability distribution of various driving intentions output by the prediction model, and selecting the driving intention with the highest probability value as the driver's driving intention. Exemplarily, a machine learning model can be used to implement the prediction model, such as a time-series model based on Long Short-Term Memory (LSTM) networks or Transformer; this application does not limit this to such models.

[0029] In one application example, multimodal driving monitoring data is input into a pre-trained intent prediction model. The intent prediction model performs a fusion analysis on the multimodal driving monitoring data and outputs a probability distribution containing various driving intentions. For example: { "Change lanes to the left": 0.85, "Keep going straight": 0.10, "Decelerate": 0.05}. Finally, the driving intention with the highest probability value is selected as the current primary prediction result.

[0030] In some embodiments, the intent prediction confidence score is a probability value output by the intent prediction model. For example, the probability of "change lanes to the left" is 85%, or 0.85. It reflects the degree to which the intent prediction model understands the driver's intent.

[0031] The score of the correlation between the driver's driving intention and Ri is closely related to the confidence level of the intention prediction. The higher the confidence level, the higher the Ri score, and vice versa.

[0032] In this embodiment, by predicting the driver's driving intention, the subsequent rendering of the instruction information can display content that matches the driving intention, thus displaying the information that the driver needs.

[0033] S102. Obtain the degree of influence of perceived objects in the vehicle's surrounding environment on the vehicle's driving task. While predicting the driver's intentions, in this embodiment, the AR-HUD system utilizes the vehicle's external sensors (such as cameras, radar, lidar, etc.) and computer vision algorithms (such as YOLO) to perceive and understand the vehicle's surrounding environment in real time, constructing a dynamic "environmental saliency map." The core of this map is to obtain the influence of each perceived object in the vehicle's surrounding environment (e.g., other vehicles, pedestrians, traffic signs, infrastructure, etc.) on the vehicle's driving task. Using these influences, the saliency of each perceived object is calculated; for example, a dynamic saliency S is calculated. This saliency S determines whether and how the perceived object is focused on and rendered by the AR-HUD system.

[0034] In some embodiments, the degree of influence of the perceived object on the vehicle's driving task includes: (1) the degree of association with the driver's driving intention, and in some embodiments, it also includes at least one of the following: (2) the degree of threat based on the motion trajectory and (3) the degree of association with navigation path information.

[0035] The following detailed explanation and illustration of the impact of acquiring the perceived object on the vehicle's driving task, using some specific embodiments, are provided.

[0036] (1) Correlation with the driver’s driving intention: This measures the correlation between the perceived objects in the vehicle’s surrounding environment and the driver’s subjective intention. It indicates the degree to which the perceived objects appear in the environment that the driver’s driving intention points to, and is proportional to the probability value of the driver’s driving intention.

[0037] In some embodiments, where the influence of a perceived object on the vehicle's driving task includes the correlation with the driver's driving intention, obtaining the influence of a perceived object in the vehicle's surrounding environment on the vehicle's driving task includes: inputting the driver's driving intention and the intention features of the perceived object into a pre-trained correlation network model, and using the output of the correlation network model as the correlation with the driver's driving intention.

[0038] In some embodiments, a correlation network model is pre-constructed, and the driver's driving intention and the intention characteristics of the perceived object, such as the environment, state, and actions of the perceived object, are used as input data to the correlation network model, outputting the correlation degree with the driver's driving intention. In some embodiments, the correlation degree with the driver's driving intention includes a specific level and a score. For example, the correlation degree with the driver's driving intention includes three levels: high correlation, medium correlation, and low correlation, and the score ranges from [0, 1].

[0039] The following examples illustrate and explain how to calculate the level and score of correlation between driver's intentions and driving intentions.

[0040] For example, the numerical range of high correlation is Ri ≈ 0.8 - 1.0, as shown in the following example: Example 1: The driver's driving intention is to change lanes to the left. The perceived objects are vehicles in the blind spot of the left rearview mirror and vehicles rapidly approaching from behind in the left lane. The driver's driving intention, the environment in which the perceived objects are located, and their actions are used as input data into a pre-built correlation network model. The output result is that it is highly correlated with the driver's driving intention, with a score of 0.9.

[0041] Example 2: The driver's driving intention is to overtake. The perceived objects are the slower vehicle directly in front and the safe space for overtaking in the left lane. The driver's driving intention, the environment of the perceived objects, and the driver's actions are used as input data to a pre-built correlation network model. The output result is that the correlation with the driver's driving intention is high, with a score of 0.9.

[0042] Example 3: The driver's driving intention is to park on the side of the road. The forward vision system identifies available parking spaces, pedestrians or other obstacles around the parking spaces, etc. The driver's driving intention, the environment in which the perceived objects are located, and their actions are used as input data into a pre-built correlation network model. The output result is that it is highly correlated with the driver's driving intention, with a score of 0.8.

[0043] For example, the numerical range for moderate correlation is Ri ≈ 0.4 - 0.7, as shown in the following examples: Example 1: The driver's driving intention is to change lanes to the left. The perceived objects are vehicles moving slowly in the distance ahead in the left lane (which can be considered potential following objects after the lane change). The driver's driving intention, the environment of the perceived objects, and their actions are used as input data into a pre-built correlation network model. The output result is that the correlation with the driver's driving intention is moderate, with a score of 0.5.

[0044] Example 2: The driver's driving intention is to follow the car in front. The driver's driving intention and the environment, state and behavior of the perceived objects are used as input data into a pre-built correlation network model. The output result is that the correlation with the driver's driving intention is moderate, with a score of 0.4.

[0045] For example, the numerical range of low correlation is Ri ≈ 0.0 - 0.3, as shown in the following examples: Example 1: The driver's driving intention is to change lanes to the left, and all vehicles traveling in the right lane or oncoming lane are perceived objects. The driver's driving intention, the environment in which the perceived objects are located, and their actions are used as input data into a pre-built correlation network model. The output result is that the correlation with the driver's driving intention is low, with a score of 0.2.

[0046] Example 2: The driver's driving intention is to keep going straight. The perceived objects such as shops on both sides of the road have low correlation with the driver's driving intention. The above driving intention, the environment in which the perceived objects are located, and the actions are used as input data into a pre-built correlation network model. The output result is a score of 0.1.

[0047] It is understood that the examples of the correlation between the driver's driving intention and the above-mentioned correlation are merely illustrative, and the embodiments of this application do not limit the specific implementation of the correlation between the driver's driving intention and the correlation.

[0048] (2) Threat level based on motion trajectory: It measures the objective danger of the perceived object in the surrounding environment to the vehicle, and indicates the degree of danger of the physical state and motion trajectory of the perceived object colliding with the vehicle.

[0049] In some embodiments, when the impact of a perceived object on the vehicle's driving task includes the threat level based on its motion trajectory, obtaining the impact of a perceived object in the vehicle's surrounding environment on the vehicle's driving task includes: inputting the vehicle's driving trajectory features and the motion trajectory features of the perceived object into a pre-trained correlation network model, and using the output of the correlation network model as the threat level based on the motion trajectory.

[0050] In some embodiments, a correlation network model is pre-constructed, and the vehicle's driving characteristics, such as the vehicle's physical state and driving status, and the motion trajectory characteristics of the perceived object, such as the perceived object's physical state and motion trajectory, are used as input data to the correlation network model, outputting a threat level based on the motion trajectory. In some embodiments, the threat level based on the motion trajectory includes a specific level and a score. For example, the threat level based on the motion trajectory includes three levels: high correlation, medium correlation, and low correlation, and the score ranges from [0, 1].

[0051] The following examples illustrate and explain how to obtain the threat level and score based on movement trajectory.

[0052] For example, the numerical range of high correlation is T ≈ 0.8 - 1.0, as shown in the following example: Example 1: The vehicle in front suddenly brakes, causing the time-to-collision (TTC) of this vehicle to drop sharply from 5 seconds to 1.5 seconds. The physical state and driving state of this vehicle and the physical state and motion trajectory of the perceived object are used as input data into a pre-built correlation network model. The output result is that the threat level of the vehicle in front is highly correlated, with a score of 0.9.

[0053] Example 2: A child suddenly runs out from between parked cars and enters the driving lane of the vehicle. The physical state and driving state of the vehicle and the physical state and motion trajectory of the perceived object are used as input data into a pre-built correlation network model. The output result is that the child's threat level is highly correlated, with a score of 0.9.

[0054] Example 3: The vehicle is driving normally in its lane. A vehicle from the side and rear merges into the lane at a higher speed without using its turn signal. The physical state and driving state of the vehicle and the physical state and trajectory of the perceived object are used as input data into a pre-built correlation network model. The output result is that the threat level of the vehicle from the side and rear is highly correlated, with a score of 0.9.

[0055] For example, the numerical range for moderate correlation is T≈0.4 - 0.7, as shown in the following examples: Example 1: This vehicle is driving normally in its lane. Vehicles in adjacent lanes are slowing down significantly but have not yet shown any intention to change lanes. The physical state and driving state of this vehicle are input into a pre-built correlation network model along with the physical state and trajectory of the perceived object. The output result is that the threat level of the vehicles in adjacent lanes is moderate, with a score of 0.7.

[0056] Example 2: When the vehicle is turning right at an intersection, a pedestrian is waiting on the crosswalk but does not step forward. The physical state and driving state of the vehicle and the physical state and motion trajectory of the perceived object are used as input data into a pre-built correlation network model. The output result is that the pedestrian's threat level is moderately correlated, with a score of 0.6.

[0057] Example 3: The vehicle is traveling normally in its lane. A bicycle is traveling along the edge of the road to the right front of the vehicle, with a slightly swaying trajectory. The physical state and driving state of the vehicle and the physical state and trajectory of the perceived object are used as input data into a pre-built correlation network model. The output result is that the threat level of the bicycle is moderately correlated, with a score of 0.6.

[0058] For example, the numerical range of low correlation is T≈0.0 - 0.3, as shown in the following example: Example 1: This vehicle is driving normally in this lane, and a vehicle is driving normally in the opposite lane. The physical state and driving state of this vehicle and the physical state and motion trajectory of the perceived object are used as input data into a pre-built correlation network model. The output result is that the threat level of the oncoming vehicle is low and the score is 0.1.

[0059] Example 2: This vehicle is driving normally in this lane. A vehicle traveling in the same direction in the distance (TTC>8 seconds) is detected. The physical state and driving state of this vehicle and the physical state and motion trajectory of the perceived object are used as input data into a pre-built correlation network model. The output result is that the threat level of the vehicle traveling in the same direction in the distance is low, with a score of 0.2.

[0060] Example 3: This vehicle is driving normally in this lane. For vehicles parked properly in roadside parking spaces, the physical state and driving state of this vehicle and the physical state and motion trajectory of the perceived object are used as input data into a pre-built correlation network model. The output result is that the threat level of the parked vehicle is low, with a score of 0.1.

[0061] It is understood that the examples of threat levels based on motion trajectories listed above are merely illustrative, and the embodiments of this application do not limit the specific implementation of threat levels based on motion trajectories.

[0062] (3) Correlation with navigation path information: This measures the importance of the perceived objects in the vehicle's surrounding environment to completing the current navigation task and indicates the degree of correlation between the perceived objects and the navigation path information.

[0063] In some embodiments, when the impact of a perceived object on the vehicle's driving task includes the threat level based on its motion trajectory, obtaining the impact of a perceived object in the vehicle's surrounding environment on the vehicle's driving task includes: inputting the vehicle's navigation path information, the identification information of the perceived object, and its location information into a pre-trained correlation network model, and using the output of the correlation network model as the correlation with the navigation path information.

[0064] In some embodiments, a correlation network model is pre-constructed, and the vehicle's navigation path information, perceived object identification information, and location are used as input data to the correlation network model, outputting the correlation degree with the navigation path information. In some embodiments, the correlation degree with the navigation path information includes a specific level and score. For example, the correlation degree with the navigation path information includes three levels: high correlation, medium correlation, and low correlation, and the score ranges from [0, 1].

[0065] The following examples illustrate and explain the levels and scores of relevance to navigation path information, using specific application examples.

[0066] For example, the numerical range of high correlation is Rp ≈ 0.8 - 1.0, as shown in the following example: Example 1: The traffic light directly above the navigation path, especially when it is red, is used as input data to a pre-built correlation network model along with the vehicle's navigation path information, the identification information of the aforementioned perceived object, and its location. The output result is that the correlation between the traffic light and the navigation path information is high, with a score of 0.9.

[0067] Example 2: When the navigation prompts "Turn right along 'XX Building'", the vision system identifies "XX Building". The navigation route information of this vehicle and "XX Building" are used as input data to input the pre-built correlation network model. The output result is that the correlation between "XX Building" and the navigation route information is high, with a score of 0.9.

[0068] Example 3: The navigation route passes through highway toll stations, especially the ETC lanes recommended by the system. The navigation route information of the vehicle and the highway toll stations and ETC lanes are used as input data into a pre-built correlation network model. The output result is that the correlation with the navigation route information is high, with a score of 0.9.

[0069] Example 4: The navigation planning route requires sensing objects such as highway exit signs. The navigation route information of the vehicle and the highway exit signs are used as input data to a pre-built correlation network model. The output result is that the correlation with the navigation route information is high, with a score of 0.9.

[0070] For example, the numerical range for moderate correlation is Rp≈0.4 - 0.7, as shown in the following examples: Example 1: The navigation route information, such as speed limit signs or section speed measurement start signs, are perceived objects. The navigation route information of the vehicle and the speed limit signs or section speed measurement start signs are used as input data to a pre-built correlation network model. The output result is that the correlation with the navigation route information is moderate, with a score of 0.7.

[0071] Example 2: The navigation map displays objects such as gas stations located ahead of the route (when the vehicle's fuel level is low). The navigation route information of the vehicle and the gas stations ahead are used as input data into a pre-built correlation network model. The output result is that the correlation with the navigation route information is moderate, with a score of 0.5.

[0072] For example, the numerical range for low correlation is Rp≈0.0 - 0.3, as shown in the following examples: Example 1: Other road signs and other perceived objects on non-navigation routes are used as input data to a pre-built correlation network model. The output result is that the correlation with the navigation route information is low, with a score of 0.0.

[0073] Example 2: For perceived objects such as commercial billboards or shop signs on both sides of the road (unless they are used as landmarks by navigation), the navigation route information of the vehicle is input into a pre-built correlation network model along with the commercial billboards or shop signs. The output result is that the correlation with the navigation route information is low, with a score of 0.1.

[0074] It is understood that the examples of the correlation between navigation path information listed above are merely illustrative, and the embodiments of this application do not limit the specific implementation of the correlation between navigation path information.

[0075] In this embodiment, the salience of each perceived object in the vehicle's surrounding environment can be obtained by weighted calculation in step S103, so that the indication information can be rendered and displayed based on the perceived objects in the current driving situation.

[0076] S103. The impact of the perceived object on the vehicle's driving task is weighted and calculated to obtain the salience of the perceived object; In some embodiments, the saliency of the perceived object is obtained by weighted calculation of the impact of the perceived object on the driving task of the vehicle. This includes: normalizing each parameter of the impact of the driving task, and then multiplying it by the weight corresponding to each parameter to obtain the saliency of the perceived object. In this embodiment, the weight corresponding to each parameter is adjusted according to the vehicle status and driving scenario.

[0077] In this embodiment, the three parameters of the driving task's impact are normalized so that their values ​​are between [0,1], facilitating a unified weighted calculation to obtain the saliency of the perceived object. In one application example, the saliency of the perceived object can be calculated using the following formula:

[0078] Where S represents the salience of the perceived object, and T represents the threat level based on the motion trajectory. To determine the degree of correlation with the driver's driving intentions, To determine the correlation with navigation path information. Based on a pre-built correlation network model, input data can be used to obtain... The scores are calculated using the weighting coefficients of the three parameters: w1, w2, and w3. The allocation of w1, w2, and w3 is dynamic, not static. The system will automatically adjust the proportion of the three according to the vehicle status and driving scenario to adapt to different driving task priorities.

[0079] For any perceived target in the vehicle's surrounding environment (whether it's a vehicle, pedestrian, or road sign), the system can calculate a unified, comprehensive saliency at any given time using the formula described above. This saliency score, S-value, is the unique quantitative representation of the perceived object's "comprehensive importance" to the driver in the current context.

[0080] Below, we will explain in detail the values ​​of the weighting coefficients (w1, w2, w3) in conjunction with some vehicle states and driving scenarios.

[0081] Scenario 1: Driving in congested urban areas Features: Close vehicle spacing, mixed pedestrian / non-motorized vehicle traffic, rapidly changing road conditions, requiring frequent acceleration and deceleration and lane keeping.

[0082] Strategy: Prioritize potential collision threats (T) and the driver's short-term following / lane-changing intentions (Ri). Navigation guidance is relatively secondary.

[0083] Weight values: w1=0.5, w2=0.4, w3=0.1.

[0084] Scenario 2: Long-distance cruise on the highway (navigation not activated) Features: High speed, long distance between vehicles, relatively simple environment, main task is to maintain lane and monitor surrounding high-speed vehicles.

[0085] Strategy: Safety is the top priority; monitoring of high-speed collision threats (T) has the highest weight. Driver intent (Ri) is primarily reflected in overtaking maneuvers. Path correlation (Rp) is 0.

[0086] Weight values: w1=0.7, w2=0.3, w3=0.0.

[0087] Scenario 3: Driving on complex overpasses or intersections using navigation Features: The route is complex with many forks, making it easy to take the wrong turn, and requires close attention to vehicles changing lanes.

[0088] Strategy: Strict adherence to navigation is the top priority, therefore the weight of path relevance (Rp) has been significantly increased. At the same time, the collision threat (T) during lane changes and intersection crossings cannot be ignored.

[0089] Weight values: w1=0.4, w2=0.2, w3=0.4 Scenario 4: Finding a parking space near the destination Features: Low-speed driving, with the driver highly focused on scanning the environment to find a specific target (parking space).

[0090] Strategy: At this point, the driver's "search" intention (Ri) becomes the dominant factor, and the system should prioritize highlighting targets related to this intention (such as empty parking spaces or obstacles).

[0091] Weight values: w1=0.3, w2=0.6, w3=0.1 It is understood that the values ​​of the weight coefficients (w1, w2, w3) for the various vehicle states and driving scenarios listed above are merely illustrative examples. The weights corresponding to each parameter in the embodiments of this application are adjusted according to the vehicle state and driving scenario, and no specific restrictions are imposed.

[0092] In this embodiment, the weight values ​​can be adjusted according to the vehicle status and driving scenario to obtain the salience of the perceived objects in the environment. Based on the salience, the importance of the perceived objects in the driving scenario can be distinguished, and the indication information can be displayed differently for the perceived objects in different driving scenarios to achieve intelligent AR rendering display.

[0093] For example, taking scenario 2 above as an example, the vehicle is traveling in its lane, and there is another vehicle traveling normally 200 meters ahead. The driver intends to overtake. Based on the pre-built correlation network model, by inputting the input data, the current scenario can be obtained. The salience of a perceived object can be calculated using the following formula:

[0094] In some embodiments, the score of the correlation between the driver's driving intention and Ri is closely related to the confidence level of the intention prediction; the higher the confidence level, the higher the Ri value, and vice versa. The calculation of the saliency of the perceived object is continuous and complete, performed on all relevant perceived objects. The core safety principle of the system is that the saliency calculation will not stop simply because the intention confidence level is low. The level of intention confidence primarily affects... The magnitude of this term modifies the importance related to the driver's subjective intent, but it never overrides objective safety (determined by the T value). Safety is always paramount. For example, if the objective threat level T of a perceived object is extremely high (e.g., an impending rear-end collision, T=1.0), then even if the driver's intent is unclear (resulting in a low Ri), it will not matter. This factor also contributes a very high score, ensuring that the salience easily exceeds the rendering threshold for emergency collision warnings, thus guaranteeing absolute driving safety. Conversely, when a strong driving intention is predicted (high confidence), the system assigns a very high score to the relevant perceived object. The value is adjusted so that its significance is more likely to exceed the rendering threshold of the corresponding indication information (such as the rendering threshold of blind spot danger warning), thereby achieving intelligent scene prompts.

[0095] S104. Based on the salience of the perceived object, display at least a portion of the indication information corresponding to the perceived object in AR form on the windshield.

[0096] In some embodiments, displaying at least a portion of the indication information corresponding to the perceived object in AR form on the windshield based on the salience of the perceived object includes: displaying at least a portion of the indication information corresponding to the perceived object in AR form on the windshield when the salience of the perceived object is greater than the rendering threshold of the indication information corresponding to the at least a portion of the perceived object.

[0097] In some embodiments, the indication information corresponding to the perceived object includes at least one of the following: (1) safety warning indication information, (2) navigation guidance indication information, (3) driving assistance indication information and (4) ecological information indication information, wherein the rendering threshold of safety warning indication information > the rendering threshold of navigation guidance indication information > the rendering threshold of driving assistance indication information > the rendering threshold of ecological information indication information.

[0098] The following detailed explanations and descriptions of the specific categories, thresholds, and rendering formats of the indication information are provided in conjunction with some specific embodiments.

[0099] (1) Safety warning information has the highest rendering threshold and the highest rendering priority.

[0100] Safety warning information may include, for example, emergency collision warning, blind spot hazard warning, lane departure warning, and pedestrian warning.

[0101] Example 1: If the collision time (TTC) between the perceived object and the vehicle is less than 2.5 seconds, an emergency collision warning is displayed. This can be done using a "red flashing box + arrow" rendering to remind the driver to avoid rear-end / side collisions.

[0102] Example 2: If the perceived object is a vehicle in the blind spot and the speed difference between the vehicle and the driver is greater than 20 km / h, a blind spot hazard warning will be displayed. This can be done by rendering the vehicle in yellow outline to remind the driver and prevent a collision when changing lanes.

[0103] Example 3: The perceived object is a vehicle in the left lane ahead that has deviated from its lane, for example, its wheels are on the lane line and it has not used its turn signal. A lane departure warning can be displayed, which can be rendered using a "red boundary line + vibration alert" to remind the driver and prevent a collision due to the vehicle ahead deviating from its lane.

[0104] Example 4: If the perceived object is a pedestrian entering a risk area, a pedestrian warning can be displayed. This can be achieved by using a rendering format of "highlighted zebra crossing + dynamic tracking box" to remind drivers that there is a pedestrian ahead and protect vulnerable road users.

[0105] (2) The rendering threshold for navigation guidance information is lower than that for safety warning information, and the rendering priority is lower than that for safety warning information.

[0106] Navigation guidance information may include, for example, turn arrows, lane-level guidance, landmark enhancements, and ETC lane instructions.

[0107] Example 1: Display a turn arrow 50 meters before the navigation waypoint. This can be rendered as a "3D floating arrow (fitted to the road surface)" to remind the driver and prevent them from missing the turn.

[0108] Example 2: 300 meters before the highway fork, lane-level guidance can be displayed, using a rendering format of "blue lane lines + exit signs" to achieve accurate guidance in complex road conditions.

[0109] Example 3: The navigation prompts "Turn right at XX Building". Enabling landmark enhancement can be achieved by using a rendering format of "glowing building outline + text label" to enhance environmental reference.

[0110] Example 4: 200 meters before the toll station, the ETC lane indicator can be displayed using a "green arrow" rendering to optimize the toll collection process and speed up passage through the toll station.

[0111] (3) The rendering threshold of driving assistance instruction information is lower than that of navigation guidance instruction information, and the rendering priority is lower than that of navigation guidance instruction information.

[0112] Driving assistance information may include, for example, adaptive cruise control, speed limit signs, and hill start assist.

[0113] Example 1: When Adaptive Cruise Control (ACC) is active, a rendering format of "vehicle distance scale + following marker" can be used to visualize the following distance.

[0114] Example 2: When a road speed limit sign is detected, the speed limit sign can be displayed. This can be done using a "floating sign + speeding red circle warning" rendering to avoid violations.

[0115] Example 3: When a road slope > 5% is detected, a slope prompt is displayed. The rendering format can be "elevation arrow + slope percentage" to predict power demand.

[0116] (4) Ecological information type indication information has the lowest rendering threshold and the lowest rendering priority.

[0117] Ecological information-related instructions may include, for example, charging station guidance and media control.

[0118] Example 1: When the electric vehicle's battery level is less than 30%, display charging station guidance. This can be achieved by using a "battery icon + distance indicator" rendering to enable timely charging and alleviate range anxiety.

[0119] Example 2: When the driver issues the voice command "Show playlist", the media control can be displayed using a "semi-transparent album art" rendering to reduce the need for central control screen operation and improve driving safety.

[0120] Figure 2 This is a schematic diagram illustrating another process for displaying indication information based on the salience of a perceived object, provided in an embodiment of this application. In some embodiments, such as Figure 2 As shown, the process includes the following steps (S201-S202): S201. Identify the target perceived object that matches the driver's driving intention from the perceived objects; For example, if the DMS detects that the driver is looking at the left rearview mirror for more than a preset time, it will analyze and predict the driver's driving intention based on the intention prediction model. The probability of the intention to "change lanes to the left" is 0.85. At this time, the vehicles, pedestrians and other objects in the left lane and blind spot are the target objects that match the driver's driving intention.

[0121] In this embodiment of the application, the sensing objects in the vehicle's surrounding environment include multiple objects. The sensing object that matches the driver's driving intention is selected as the target sensing object for rendering instruction information. This can further filter out more important information for display, thereby avoiding safety hazards caused by visual interference to the driver due to information overload.

[0122] S202. When the salience of the target perception object is greater than the rendering threshold of the indication information corresponding to at least some of the target perception objects, the indication information corresponding to at least some of the target perception objects is displayed in AR form on the windshield.

[0123] In some embodiments, an indication is triggered and rendered onto the corresponding sensing object only when the saliency score of a sensing object is greater than the rendering threshold of an indication. A sensing object may have multiple indications superimposed on it because its S value exceeds multiple rendering thresholds simultaneously.

[0124] In one application example, the rendering threshold is set to indicate the following information: The rendering threshold for emergency collision warning is 0.90 (extremely high, triggered only in extremely critical situations); The rendering threshold for blind spot hazard warning is 0.75 (relatively high, requiring a clear intent and risk to trigger). The rendering threshold for highlighting navigation landmarks is 0.60 (medium, triggered when highly relevant to navigation tasks); The rendering threshold for the standard rate limit warning is 0.40 (low, as a general reminder); If the salience of a certain perceived object is 0.8, which is greater than the rendering threshold of blind spot hazard warning, navigation landmark highlighting, and regular speed limit warning, then all three types of indication information will be triggered and rendered onto the corresponding perceived object.

[0125] In some embodiments, the perceived objects include multiple objects; for example, other vehicles, pedestrians, traffic signs, infrastructure, etc., in the environment surrounding the vehicle. In some embodiments, rendering the indication information includes: sorting the salience of the perceived objects, and according to the order of salience from high to low, decreasing the rendering display level of the indication information corresponding to the perceived objects in sequence, and displaying at least some of the indication information corresponding to the perceived objects in AR form on the windshield. In this embodiment, when multiple perceived objects simultaneously meet the rendering conditions, that is, when the salience of multiple perceived objects exceeds the rendering threshold of their respective indication information, the perceived object with higher salience has a higher rendering priority and will be presented in a more eye-catching, dynamic, and visually central manner. Thus, perceived objects that have a greater impact on the driver's driving intentions can be highlighted, emphasizing the information most needed by the driver, helping the driver distinguish between primary and secondary information, and providing a more intelligent AR-HUD information rendering method.

[0126] In some embodiments, displaying at least a portion of the indication information corresponding to the perceived object in AR form on the windshield includes: enhancing the display of the safety warning indication information in the indication information when the indication information corresponding to at least a portion of the perceived object includes safety warning indication information and the rendering threshold of the safety warning indication information is the threshold of the highest security level, and suppressing the display of other indication information in the indication information corresponding to at least a portion of the perceived object.

[0127] In this embodiment, when the salience of any perceived object exceeds the rendering threshold of the highest safety level (such as the rendering threshold of an emergency collision warning), the system immediately and forcibly enhances the display of the highest safety level indication information and suppresses the display of all non-safety-related secondary information (such as media information and telephone alerts) by setting the transparency to the minimum or hiding it, ensuring that the driver's attention can be 100% focused on the most dangerous event. For example, if the collision time (TTC) between a perceived object and the vehicle is less than 2.5 seconds, and its calculated salience is 0.95, which is greater than the rendering threshold of an emergency collision warning (0.90), then the emergency collision warning is triggered and enhanced on the perceived object, while the display of other indication information that is not at the highest safety level is suppressed. This allows the driver to be aware of the most dangerous information in the vehicle's surrounding environment in a timely manner and avoid danger.

[0128] To more clearly illustrate the technical solutions provided in the embodiments of this application, the following uses "intelligent lane change warning" as an example, combined with Figure 3, to further illustrate a method for displaying indication information provided in this application. Figure 3 This is a schematic flowchart of a method for displaying indication information provided in an embodiment of this application.

[0129] like Figure 3 As shown, the method includes the following steps (S301-S307): S301. Acquire multimodal driving monitoring data of the vehicle and the degree of influence of perceived objects in the vehicle's surrounding environment on the vehicle's driving tasks. For example, the DMS driver monitoring data shows that the eye is focused on the left rearview mirror, and the navigation route information shows "turn left at the intersection ahead".

[0130] S302. Input the multimodal driving monitoring data into the pre-trained intention prediction model to obtain the intention probability value of "change lanes to the left" as 0.85; S303, Identify the target vehicle in the left blind spot as the target perception object; S304. Using the formula above, the salience of the perceived object is obtained by weighting the impact of the target perceived object on the driving task of the vehicle. S305. Determine whether the salience of the perceived object is greater than the rendering threshold of the blind spot danger warning. If it is greater, proceed to step S306; otherwise, proceed to step S307. S306: Highlight the blind spot vehicle warning while suppressing the display of secondary information; S307, Maintain standard HUD display.

[0131] The method for displaying instruction information provided in this embodiment can render instruction information based on the predicted driving intention and the current driving situation, thereby achieving intelligent instruction information rendering and displaying content that matches the driving intention and the current driving situation. This provides the driver with the most needed information and avoids safety hazards caused by visual interference due to information overload.

[0132] An exemplary embodiment of this application provides a display device 100 for indicating information. Figure 4 A structural block diagram of a display device for indicating information provided in an exemplary embodiment of this application is shown. The aforementioned display device for indicating information is applied to a vehicle and can achieve the following: Figures 1 to 3All or part of the contents of any of the illustrated embodiments. The following is only a brief description of the structure and function of the display device for the indication information; for other matters not covered herein, please refer to the relevant descriptions in the above-described method for displaying indication information. The embodiments of the display device for the indication information correspond to the embodiments of the above-described method for displaying indication information. All implementation processes and methods of the above-described method embodiments can be applied to the embodiments of the display device for the indication information and can achieve the same technical effects.

[0133] like Figure 4 As shown, the display device 100 for the indication information includes: a driving intention prediction module 101, a perception object analysis module 102, and a display module 103. In this embodiment, the driving intention prediction module 101 is used to acquire multimodal driving monitoring data and predict the driver's driving intention based on the multimodal driving monitoring data. The vehicle multimodal monitoring data includes one of the following: driver monitoring data, vehicle control signals, and navigation path information. The perception object analysis module 102 is used to calculate the influence of perception objects in the vehicle's surrounding environment on the vehicle's driving task, and to perform a weighted calculation on the influence of perception objects on the vehicle's driving task to obtain the salience of the perception objects. The influence of perception objects on the vehicle's driving task includes: the correlation with the driver's driving intention, and the influence of perception objects on the vehicle's driving task also includes at least one of the following: the threat level based on the motion trajectory and the correlation with the navigation path information. The display module 103 is used to display at least a portion of the indication information corresponding to the perception objects in AR form on the windshield based on the salience of the perception objects.

[0134] In some embodiments, the multimodal monitoring data includes one of the following: (1) driver monitoring data, (2) vehicle control signals, and (3) navigation path information. For a detailed description of the vehicle multimodal monitoring data in this embodiment, please refer to the relevant description in the above-mentioned method for displaying instruction information; it will not be repeated here. In this embodiment, the above-mentioned vehicle multimodal monitoring data can be used to predict the driver's driving intentions, so that the subsequent rendering of instruction information can be based on the driver's driving intentions.

[0135] In some embodiments, the driving intention prediction module 101 predicts the driver's driving intention based on multimodal driving monitoring data in the following manner: The multimodal driving monitoring data is input into a pre-trained intention prediction model to obtain the probability distribution of various driving intentions output by the intention prediction model, and the driving intention with the highest probability value is selected as the driver's driving intention. In this embodiment, by predicting the driver's driving intention, the subsequent rendering of instruction information can display content matching the driving intention, thus displaying the information needed by the driver.

[0136] In some embodiments, the degree of influence of the perceived object on the vehicle's driving task includes: (1) the degree of association with the driver's driving intention, and in some embodiments, it also includes at least one of the following: (2) the degree of threat based on the motion trajectory and (3) the degree of association with navigation path information.

[0137] The threat level based on the motion trajectory indicates the physical state of the perceived object and the degree of danger of the collision between the motion trajectory and the vehicle; the correlation with the driver's driving intention indicates the degree to which the perceived object appears in the environment pointed to by the driver's driving intention, and is proportional to the probability value of the driver's driving intention; the correlation with navigation path information indicates the degree of correlation between the perceived object and the navigation path information.

[0138] The perception object analysis module 102 obtains the impact of perceived objects in the vehicle's surrounding environment on the vehicle's driving task in the following ways: The driver's driving intention and the intention features of the perceived object are input into a pre-trained correlation network model, and the output of the correlation network model is used as the correlation degree with the driver's driving intention. When the impact of the perceived object on the vehicle's driving task includes the threat level based on the motion trajectory, the vehicle's driving trajectory features and the perceived object's motion trajectory features are input into a pre-trained correlation network model, and the output of the correlation network model is used as the threat level based on the motion trajectory. When the impact of the perceived object on the vehicle's driving task includes its correlation with navigation path information, the vehicle's navigation path information, the identification information of the perceived object, and its location information are input into a pre-trained correlation network model, and the output of the correlation network model is used as the correlation with the navigation path information.

[0139] For a description of the impact of the perceived objects on the vehicle's driving task in this embodiment, please refer to the relevant explanation in the above-mentioned method for displaying instruction information; it will not be repeated here. In this embodiment, the salience of the perceived objects can be obtained by weighted calculation using the impact of each perceived object in the vehicle's surrounding environment on the vehicle's driving task, so that the instruction information can be rendered based on the perceived objects in the current driving context.

[0140] In some embodiments, the perception object analysis module 103 calculates the weighted impact of the perception object on the vehicle's driving task in the following way to obtain the saliency of the perception object: after normalizing each parameter of the driving task impact, it multiplies it by the weight corresponding to each parameter and calculates the weighted impact to obtain the saliency of the perception object. The weight corresponding to each parameter is adjusted according to the vehicle's state and driving scenario.

[0141] In this embodiment, the three parameters of the driving task's impact are normalized so that their values ​​are between [0,1], facilitating a unified weighted calculation to obtain the saliency of the perceived object. In one application example, the saliency of the perceived object can be calculated using the following formula:

[0142] Where S represents the salience of the perceived object, and T represents the threat level based on the motion trajectory. To determine the degree of correlation with the driver's driving intentions, To determine the correlation with navigation path information. Based on a pre-built correlation network model, input data can be used to obtain... The scores are calculated using the weighting coefficients of the three parameters: w1, w2, and w3. The allocation of w1, w2, and w3 is dynamic, not static. The system automatically adjusts the weight of these three factors based on vehicle status and driving scenario to adapt to different driving task priorities. For a detailed description of the weighting coefficients in this embodiment, please refer to the relevant explanations in the above-mentioned display method for indicator information; they will not be repeated here.

[0143] In this embodiment, the weight values ​​can be adjusted according to the vehicle's status and driving scenario to obtain the salience of the perceived objects in the environment. Based on the salience, the importance of the perceived objects in the driving scenario can be distinguished, and the instruction information can be rendered differently for the perceived objects in different driving scenarios to achieve intelligent AR rendering.

[0144] In some embodiments, the score of the correlation between the driver's driving intention and Ri is closely related to the confidence level of the intention prediction; the higher the confidence level, the higher the Ri value, and vice versa. The calculation of the saliency of the perceived object is continuous and complete, performed on all relevant perceived objects. The core safety principle of the system is that the saliency calculation will not stop simply because the intention confidence level is low. The level of intention confidence primarily affects... The magnitude of this term modifies the importance related to the driver's subjective intent, but it never overrides objective safety (determined by the T value). Safety is always paramount. For example, if the objective threat level T of a perceived object is extremely high (e.g., an impending rear-end collision, T=1.0), then even if the driver's intent is unclear (resulting in a low Ri), it will not matter. This factor also contributes a very high score, ensuring that the salience easily exceeds the rendering threshold for emergency collision warnings, thus guaranteeing absolute driving safety. Conversely, when a strong driving intention is predicted (high confidence), the system assigns a very high score to the relevant perceived object. The value is adjusted so that its significance is more likely to exceed the rendering threshold of the corresponding indication information (such as the rendering threshold of blind spot danger warning), thereby achieving intelligent scene prompts.

[0145] In some embodiments, the display module 103 displays at least a portion of the indication information corresponding to the perceived object in AR form on the windshield based on the salience of the perceived object: when the salience of the perceived object is greater than the rendering threshold of the indication information corresponding to at least a portion of the perceived object, the indication information corresponding to at least a portion of the perceived object is displayed in AR form on the windshield.

[0146] In some embodiments, the indication information corresponding to the perceived object includes at least one of the following: (1) safety warning indication information, (2) navigation guidance indication information, (3) driving assistance indication information and (4) ecological information indication information, wherein the rendering threshold of safety warning indication information > the rendering threshold of navigation guidance indication information > the rendering threshold of driving assistance indication information > the rendering threshold of ecological information indication information.

[0147] For a detailed description of the specific categories, thresholds, and rendering formats of the indication information in this embodiment, please refer to the relevant descriptions in the above-mentioned method for displaying indication information; these details will not be repeated here.

[0148] In some embodiments, an indication is only triggered and rendered onto the corresponding sensing object when its salience is greater than a rendering threshold. A sensing object may have multiple indications superimposed on it because its salience exceeds multiple rendering thresholds simultaneously.

[0149] In one application example, the rendering threshold is set to indicate the following information: The rendering threshold for emergency collision warning is 0.90 (extremely high, triggered only in extremely critical situations); The rendering threshold for blind spot hazard warning is 0.75 (relatively high, requiring a clear intent and risk to trigger). The rendering threshold for highlighting navigation landmarks is 0.60 (medium, triggered when highly relevant to navigation tasks); The rendering threshold for the standard rate limit warning is 0.40 (low, as a general reminder); If the salience of a certain perceived object is 0.8, which is greater than the rendering threshold of blind spot hazard warning, navigation landmark highlighting, and regular speed limit warning, then all three types of indication information will be triggered and rendered onto the corresponding perceived object.

[0150] In some embodiments, the sensed objects include multiple objects; for example, other vehicles, pedestrians, traffic signs, infrastructure, etc., in the environment surrounding the vehicle. In some embodiments, the display module 103 renders the indication information in the following manner: sorting the salience of the sensed objects, and decreasing the rendering display level of the indication information corresponding to the sensed objects in descending order of salience, and displaying at least some of the indication information corresponding to the sensed objects in AR form on the windshield. In this embodiment, when multiple sensed objects simultaneously meet the rendering conditions (i.e., the salience of multiple sensed objects exceeds the rendering threshold of their respective indication information), the sensed object with higher salience has a higher rendering priority and will be presented in a more eye-catching, dynamic, and visually central manner. Thus, sensed objects that have a greater impact on the driver's driving intentions can be highlighted, emphasizing the information most needed by the driver, helping the driver distinguish between primary and secondary information, and providing a more intelligent AR-HUD information rendering method.

[0151] In some embodiments, the display module 103 displays at least a portion of the indication information corresponding to the perceived objects on the windshield in AR form by: enhancing the display of the safety warning indication information when the indication information corresponding to at least a portion of the perceived objects includes safety warning indication information, and the rendering threshold of the safety warning indication information is the highest safety level threshold; and suppressing the display of other indication information. In this embodiment, when the salience of any perceived object exceeds the highest safety level rendering threshold (such as the rendering threshold of an emergency collision warning), the display module 103 immediately and forcibly enhances the highest safety level indication information and suppresses the display of all non-safety-related secondary information (such as media information and telephone reminders) by adjusting the transparency to the minimum or hiding it, to ensure that the driver's attention can be 100% focused on the most dangerous event. For example, if the collision time (TTC) between a perceived object and the vehicle is less than 2.5 seconds, and its calculated salience is 0.95, which is greater than the rendering threshold (0.90) for emergency collision warning, then the emergency collision warning is triggered and enhanced on the perceived object. Information display suppression is applied to non-highest safety level indications. This allows the driver to promptly understand the most dangerous information in the vehicle's surroundings and avoid potential hazards.

[0152] The indicator information display device provided in this embodiment can render the indicator information based on the predicted driving intention of the driver and the current driving situation, realize intelligent AR rendering, display content that matches the driving intention and the current driving situation, provide the driver with the most needed information, and thus avoid the safety hazards caused by visual interference to the driver due to information overload.

[0153] Regarding the apparatus in the above embodiments, the specific manner in which each unit performs its operation has been described in detail in the embodiments related to the method, and will not be elaborated upon here.

[0154] Figure 5 is a structural schematic diagram of a vehicle provided in an embodiment of this application.

[0155] For example, as shown in FIG5, the vehicle includes a memory 501 and a processor 502, wherein the memory 501 stores executable program code 5011, and the processor 502 is used to call and execute the executable program code 5011 to execute the method of displaying instruction information.

[0156] This embodiment can divide the vehicle into functional modules according to the above method example. For example, each function can be assigned to a separate module, or two or more functions can be integrated into one processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods.

[0157] When each functional module is divided according to its corresponding function, the vehicle may include: a driving intention prediction module, a perception object analysis module, and a display module, etc. It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional descriptions of the corresponding functional modules, and will not be repeated here.

[0158] The vehicle provided in this embodiment is used to execute the above-described method for displaying instruction information, and thus can achieve the same effect as the above-described implementation method.

[0159] When using integrated units, the vehicle may include a processing module and a storage module. The processing module is used to control and manage the vehicle's actions. The storage module supports the vehicle in executing program code and data.

[0160] The processing module may be a processor or a controller, which can implement or execute various exemplary logic blocks, modules, and circuits as disclosed in this application. The processor may also be a combination of components that implement computational functions. For example, it may include one or more microprocessor combinations, digital signal processing (DSP) and microprocessor combinations, etc., and the storage module may be a memory.

[0161] This embodiment also provides a computer-readable storage medium (including but not limited to disk storage, CD-ROM, optical storage, etc.) storing computer program code. When the computer program code is run on a computer, the computer executes the above-described related method steps to implement the method for displaying indication information provided in the above embodiment.

[0162] This embodiment also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned related steps to implement a method for displaying instruction information provided in the above embodiment.

[0163] The beneficial effects of the above embodiments can be referred to the beneficial effects of the corresponding methods provided above, and will not be repeated here.

[0164] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0165] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0166] In the description of this application, it should be understood that if the terms "upper", "lower", "front", "rear", "left" and "right" are used to indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, they are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the position or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this application.

[0167] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes the element.

[0168] The above are merely embodiments of this application and are not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A method for displaying indicator information, characterized in that, The method includes: Acquire multimodal driving monitoring data and predict the driver's driving intention based on the multimodal driving monitoring data. The multimodal monitoring data includes one of the following: driver monitoring data, vehicle control signals, and navigation path information. The influence of perceived objects in the vehicle's surrounding environment on the vehicle's driving task is obtained. The influence of the perceived objects on the vehicle's driving task includes: the degree of correlation with the driver's driving intention. The influence of the perceived objects on the vehicle's driving task also includes at least one of the following: the degree of threat based on the movement trajectory and the degree of correlation with the navigation path information. The saliency of the perceived object is obtained by weighting the impact of the perceived object on the driving task of the vehicle. Based on the salience of the perceived object, at least a portion of the indication information corresponding to the perceived object is displayed in AR form on the windshield.

2. The method according to claim 1, characterized in that, The driver monitoring data includes one of the following: the driver's head posture, hand movements, body posture, and eye gaze point; The vehicle control signals include one of the following: steering wheel angle, turn signal status, and input mode of acceleration / deceleration pedals; The step of predicting the driver's driving intention based on the multimodal driving monitoring data includes: The multimodal driving monitoring data is input into a pre-trained intention prediction model to obtain the probability distribution of various driving intentions output by the intention prediction model. The driving intention with the highest probability value is selected as the driver's driving intention.

3. The method according to claim 1, characterized in that, The acquisition of the influence of perceived objects in the vehicle's surrounding environment on the vehicle's driving task includes: The driver's driving intention and the intention features of the perceived object are input into a pre-trained correlation network model, and the output of the correlation network model is used as the correlation degree with the driver's driving intention. When the impact of the perceived object on the vehicle's driving task includes the threat level based on the motion trajectory, the driving trajectory features of the vehicle and the motion trajectory features of the perceived object are input into a pre-trained correlation network model, and the output of the correlation network model is used as the threat level based on the motion trajectory. When the influence of the perceived object on the vehicle's driving task includes the correlation with the navigation path information, the vehicle's navigation path information, the identification information of the perceived object, and its location information are input into a pre-trained correlation network model, and the output of the correlation network model is used as the correlation with the navigation path information.

4. The method according to any one of claims 1 to 3, characterized in that, The weighted calculation of the impact of the perceived object on the driving task of the vehicle to obtain the salience of the perceived object includes: After normalizing each parameter of the influence of the driving task, the salience of the perceived object is obtained by multiplying it by the weight corresponding to each parameter and then performing a weighted calculation. The weights corresponding to each parameter are adjusted according to the vehicle status and driving scenario.

5. The method according to claim 1, characterized in that, The step of displaying at least a portion of the indication information corresponding to the perceived object in AR format on the windshield based on the salience of the perceived object includes: When the salience of the perceived object is greater than the rendering threshold of the indication information corresponding to at least a portion of the perceived object, the indication information corresponding to at least a portion of the perceived object is displayed in AR form on the windshield. The indication information corresponding to the perceived object includes at least one of the following: safety warning indication information, navigation guidance indication information, driving assistance indication information, and ecological information indication information. The rendering threshold of the safety warning indication information is greater than the rendering threshold of the navigation guidance indication information, the rendering threshold of the driving assistance indication information, and the rendering threshold of the ecological information indication information.

6. The method according to claim 5, characterized in that, The sensing objects include multiple ones; The step of displaying at least a portion of the indicated information corresponding to the sensed objects in AR format on the windshield includes: The salience of the perceived objects is sorted, and the rendering display degree of the corresponding indication information of the perceived objects is reduced in descending order of salience. At least part of the indication information of the perceived objects is displayed in AR form on the windshield.

7. The method according to claim 5, characterized in that, The step of displaying at least a portion of the indicated information corresponding to the sensed objects in AR format on the windshield includes: When at least some of the indicated information corresponding to the perceived objects includes safety warning type indicated information, and the rendering threshold of the safety warning type indicated information is the threshold of the highest security level, the safety warning type indicated information is enhanced and other indicated information in at least some of the indicated information corresponding to the perceived objects is suppressed.

8. A display device for indicating information, characterized in that, include: The driving intention prediction module is used to acquire multimodal driving monitoring data and predict the driver's driving intention based on the multimodal driving monitoring data. The vehicle multimodal monitoring data includes one of the following: driver monitoring data, vehicle control signals, and navigation path information. The perception object analysis module is used to calculate the influence of perception objects in the vehicle's surrounding environment on the vehicle's driving task, and to perform a weighted calculation on the influence of the perception objects on the vehicle's driving task to obtain the salience of the perception objects; wherein, the influence of the perception objects on the vehicle's driving task includes: the correlation with the driver's driving intention, and the influence of the perception objects on the vehicle's driving task also includes at least one of the following: the threat level based on the movement trajectory and the correlation with the navigation path information; The display module is used to display at least a portion of the indication information corresponding to the perceived object in AR form on the windshield based on the salience of the perceived object.

9. A vehicle, characterized in that, The vehicle includes at least one processor, which, when executed, implements the steps of the method for displaying indication information as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the steps of the method for displaying indication information as described in any one of claims 1 to 7.

Images