Augmented reality head-up display method and device for vehicle auxiliary element information

Abstract

The invention discloses an augmented reality head-up display method and device for vehicle auxiliary element information. The method comprises the steps of obtaining signal data recognized by a vehicle through a sensor; if it is judged that the signal data indicates that the vehicle is in the low-visibility environment at present, the current running state of the augmented reality display client is determined; if the client is in the closed state, starting an augmented reality display client and entering an augmented reality auxiliary element prompt mode for display; or, if the client is operated and is in the first display mode state, switching the first display mode state into the augmented reality auxiliary element prompt mode and displaying the augmented reality auxiliary element prompt mode; in the augmented reality auxiliary element prompting mode, acquiring environment sensing data of the vehicle; determining auxiliary element information based on the environment perception data; and displaying the auxiliary element information at a corresponding position in the augmented reality head-up display view field range. The technical problem that the view of a road in front of a vehicle is fuzzy in a low-visibility environment in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of vehicle technology, and in particular to an augmented reality head-up display method and device for vehicle auxiliary element information. Background Art

[0002] When a vehicle is driving in a low-visibility environment, visibility is reduced and the driving line of sight is severely obstructed, increasing the risk and difficulty of driving. Figure 1 is a schematic diagram of the front vehicle and lane lines in a low visibility environment in the prior art, such as Figure 1 As shown in the figure, due to the serious obstruction of vision in low-visibility environment, the driver cannot clearly distinguish the safe distance from the vehicle in front, which may cause the driver to have a poor grasp of the following distance, thus causing safety risks. In addition, due to the low visibility in low-visibility environment, the road edge line of the lane where the vehicle is located and the lane lines of other roads are blurred, which in turn causes the driver to have the risk of lane deviation while driving. In addition, since the rearview mirror is also affected by the low visibility environment and becomes unclear, the driver cannot accurately judge the safe distance between the vehicle and the rear vehicle through the rearview mirror when changing lanes, thus causing safety risks.

[0003] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0004] The embodiments of the present invention provide an augmented reality head-up display method and device for vehicle auxiliary element information, so as to at least solve the technical problem in the related art that the view of the road ahead of the vehicle is blurred in a low visibility environment.

[0005] According to one aspect of an embodiment of the present invention, there is provided an augmented reality head-up display method for auxiliary element information of a vehicle, comprising: obtaining signal data recognized by a sensor of the vehicle; if it is determined that the signal data indicates that the vehicle is currently in a low visibility environment, determining the current operating state of an augmented reality display client; if the augmented reality display client is in an off state, turning on the augmented reality display client and directly entering an augmented reality auxiliary element prompt mode for display; or, if the augmented reality display client is already running and in a first display mode state, switching the first display mode state to an augmented reality auxiliary element prompt mode and displaying it; in the augmented reality auxiliary element prompt mode, obtaining environmental perception data of the vehicle; determining auxiliary element information based on the environmental perception data; and displaying the auxiliary element information at a corresponding position within the field of view of the augmented reality head-up display, wherein the auxiliary element information includes a prompt image of an object to be prompted.

[0006] Furthermore, the auxiliary element information also includes a lane line image of the current lane where the vehicle is located. Determining the auxiliary element information based on the environmental perception data includes: acquiring the lane line image of the current lane based on the environmental perception data, and rendering the lane line image of the current lane at a corresponding position within the field of view of the augmented reality head-up display.

[0007] Furthermore, the objects to be prompted include static objects or dynamic objects, the dynamic objects include at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian or an animal within the field of view of the augmented reality head-up display of the vehicle, and the static objects include at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian or an animal within the field of view of the augmented reality head-up display of the vehicle.

[0008] Furthermore, the auxiliary element information also includes a lane line image of the lane to be traveled, and the lane line image of the lane to be traveled includes the lane line image of the current lane where the vehicle is located and the lane line image of the lane to which the vehicle is about to change lanes. Determining the auxiliary element information based on the environmental perception data also includes: obtaining lane-level light blanket control line data in a map coordinate system from a cloud server; determining the lane to be traveled of the vehicle based on the lane-level light blanket control line data, wherein the lane to be traveled includes the current lane where the vehicle is located and the lane to which the vehicle is about to change lanes; determining first lane line element information corresponding to the lane to be traveled from the environmental perception data; converting the first lane line element information from the world coordinate system to the vehicle coordinate system to obtain second lane line element information in the vehicle coordinate system; fusing the second lane line element information with the lane-level light blanket control line data to obtain third lane line element information in the vehicle coordinate system; and determining the lane line image of the lane to be traveled based on the third lane line element information.

[0009] Furthermore, the auxiliary element information also includes a lane line image of the lane to be traveled, and the lane line image of the lane to be traveled includes the lane line image of the current lane where the vehicle is located and the lane line image of the lane to which the vehicle is about to change lanes. Determining the auxiliary element information based on the environmental perception data also includes: calculating the lane-level light blanket control line data in the vehicle coordinate system according to the navigation direction data and the environmental perception data; determining the lane to be traveled of the vehicle based on the lane-level light blanket control line data, wherein the lane to be traveled includes the current lane where the vehicle is located and the lane to which the vehicle is about to change lanes; determining the lane line element information corresponding to the lane to be traveled from the environmental perception data; and determining the lane line image of the lane to be traveled based on the lane line element information.

[0010] Furthermore, the signal data includes external sensor data of the vehicle and internal sensor data of the vehicle, and the method also includes: if the external sensor data of the vehicle indicates that the vehicle is currently in at least one of a heavy rain environment, a smoke environment, a haze environment, a blizzard environment, a sandstorm environment, and a strong light environment, then it is determined that the vehicle is currently in a low visibility environment; if the internal sensor data of the vehicle indicates that the front windshield window of the vehicle is fogged up or the front windshield is suddenly blocked, then it is determined that the vehicle is currently in a low visibility environment.

[0011] Furthermore, the auxiliary element information also includes at least one of the following: a road edge line image, an image of a lane change warning light of an object to be prompted, a bird's-eye view image of the vehicle itself, and a side and rear steering image, wherein the lane change warning image of the object to be prompted is the image information of the lane change warning light of the vehicle in front, the bird's-eye view image of the vehicle itself is the image information from the bird's-eye view perspective of the vehicle itself, and the side and rear steering image is the image information from the side and rear of the vehicle itself.

[0012] Furthermore, displaying auxiliary element information at a corresponding position within the field of view of the augmented reality head-up display includes: obtaining a first position of an object to be prompted ahead in a current frame; estimating a position of the object to be prompted ahead in at least one next frame in a world coordinate system based on a relative speed between the object to be prompted ahead and the vehicle, to obtain at least one second position; performing window averaging on at least one second position to obtain a third position; converting the third position to the vehicle coordinate system to obtain a fourth position; and displaying a prompt image of the object to be prompted or a lane change warning light image of the object to be prompted at the fourth position within the field of view of the augmented reality head-up display.

[0013] Furthermore, the augmented reality head-up display method of vehicle auxiliary element information also includes: determining first image information of at least one object to be prompted in front of the vehicle, the relative delay corresponding to the first image information, and the relative speed vector between at least one object to be prompted and the vehicle based on environmental perception data, wherein the first image information is the image information of at least one object to be prompted in the vehicle coordinate system, and the relative delay includes the dynamic delay of the first image information during the acquisition, transmission and calculation process, and the fixed delay during the rendering and display process; based on the relative speed vector and the relative delay, the first image information is delayed compensated to obtain second image information, so as to display the second image information at a corresponding position within the field of view of the augmented reality head-up display, wherein the second image information is the image information of at least one object to be prompted in the vehicle coordinate system.

[0014] Furthermore, determining the relative speed vector between at least one object to be prompted and the vehicle based on the environmental perception data includes: determining first driving data of the vehicle and second driving data of at least one object to be prompted based on the environmental perception data, wherein the first driving data includes the current speed and heading angle of the vehicle in the vehicle coordinate system, and the second driving data includes the current speed and driving direction of at least one object to be prompted in the world coordinate system; converting the first driving data to the world coordinate system to obtain third driving data; determining a first relative speed vector between the vehicle and at least one object to be prompted in the world coordinate system based on the second driving data and the third driving data; converting the first relative speed vector to the vehicle coordinate system to obtain a relative speed vector, wherein the relative speed vector includes relative speed information and relative direction information.

[0015] Furthermore, based on the relative speed vector and the relative time delay, the first image information is delayed compensated to obtain the second image information, including: determining the relative distance between the vehicle and at least one object to be prompted based on the relative speed vector and the relative time delay; based on the relative distance and relative direction information, the first image information is delayed compensated in the vehicle coordinate system to obtain the second image information, wherein the delay compensation is used to adjust the coordinates of the first image information to obtain the second image information.

[0016] Furthermore, the augmented reality head-up display method of vehicle auxiliary element information also includes: if the signal data is in a target state, recording the duration of the signal data in the target state; in response to the duration being greater than a preset duration, determining that the vehicle is currently in a low visibility environment.

[0017] Further, the signal data includes a wiper activation signal, a wiper operating speed, a fog lamp activation signal, and rain light amount sensor data. The method also includes: in response to the wiper activation signal being in an activated state, the wiper operating speed being greater than a preset speed, the fog lamp activation signal being in an activated state, and the rain light amount sensor data indicating that the rainfall is greater than a preset rainfall amount and the light intensity is greater than a preset light intensity, determining that the signal data is in a target state.

[0018] Furthermore, obtaining signal data recognized by the vehicle through the sensor includes: obtaining first signal data collected by the vehicle through at least one sensor; and fusing the first signal data to obtain signal data.

[0019] According to another aspect of an embodiment of the present invention, an augmented reality head-up display device for vehicle auxiliary element information is also provided, including: a first acquisition module, used to acquire signal data recognized by the vehicle through a sensor; a first determination module, used to determine the current operating state of the augmented reality display client if it is determined that the signal data indicates that the vehicle is currently in a low visibility environment; a control module, used to turn on the augmented reality display client if the augmented reality display client is in an off state, and directly enter the augmented reality auxiliary element prompt mode for display; or, if the augmented reality display client is already running and in a first display mode state, switch the first display mode state to an augmented reality auxiliary element prompt mode and display it; a second acquisition module, used to acquire the environmental perception data of the vehicle in the augmented reality auxiliary element prompt mode; a second determination module, used to determine the auxiliary element information based on the environmental perception data; a display module, used to display the auxiliary element information at a corresponding position within the field of view of the augmented reality head-up display, wherein the auxiliary element information includes a prompt image of the object to be prompted.

[0020] Furthermore, the auxiliary element information also includes a lane line image of the current lane in which the vehicle is located. The second acquisition module is also used to acquire the lane line image of the current lane based on the environmental perception data, and render the lane line image of the current lane at a corresponding position within the field of view of the augmented reality head-up display.

[0021] Furthermore, the objects to be prompted include static objects or dynamic objects, the dynamic objects include at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian or an animal within the field of view of the augmented reality head-up display of the vehicle, and the static objects include at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian or an animal within the field of view of the augmented reality head-up display of the vehicle.

[0022] Furthermore, the auxiliary element information also includes a lane line image of the lane to be traveled, and the lane line image of the lane to be traveled includes the lane line image of the current lane where the vehicle is located and the lane line image of the lane to which the vehicle is about to change lanes. The second acquisition module is also used to obtain lane-level light carpet control line data in the map coordinate system from the cloud server; determine the lane to be traveled by the vehicle based on the lane-level light carpet control line data, wherein the lane to be traveled includes the current lane where the vehicle is located and the lane to which the vehicle is about to change lanes; determine first lane line element information corresponding to the lane to be traveled from the environmental perception data; convert the first lane line element information from the world coordinate system to the vehicle coordinate system to obtain second lane line element information in the vehicle coordinate system; fuse the second lane line element information and the lane-level light carpet control line data to obtain third lane line element information in the vehicle coordinate system; and determine the lane line image of the lane to be traveled based on the third lane line element information.

[0023] Furthermore, the auxiliary element information also includes a lane line image of the lane to be traveled, and the lane line image of the lane to be traveled includes the lane line image of the current lane where the vehicle is located and the lane line image of the lane to which the vehicle is about to change lanes. The second acquisition module is also used to calculate the lane-level light blanket control line data in the vehicle coordinate system according to the navigation direction data and the environmental perception data; determine the lane to be traveled by the vehicle based on the lane-level light blanket control line data, wherein the lane to be traveled includes the current lane where the vehicle is located and the lane to which the vehicle is about to change lanes; determine the lane line element information corresponding to the lane to be traveled from the environmental perception data; and determine the lane line image of the lane to be traveled based on the lane line element information.

[0024] Furthermore, the signal data includes external sensor data of the vehicle and internal sensor data of the vehicle. The first determination module is also used to determine that the vehicle is currently in a low visibility environment if the external sensor data of the vehicle indicates that the vehicle is currently in at least one of a heavy rain environment, a smoke environment, a haze environment, a blizzard environment, a sandstorm environment, and a strong light environment; and to determine that the vehicle is currently in a low visibility environment if the internal sensor data of the vehicle indicates that the front windshield window of the vehicle is fogged or the front windshield is suddenly blocked.

[0025] Furthermore, the auxiliary element information also includes at least one of the following: a road edge line image, an image of a lane change warning light of an object to be prompted, a bird's-eye view image of the vehicle itself, and a side and rear steering image, wherein the lane change warning image of the object to be prompted is the image information of the lane change warning light of the vehicle in front, the bird's-eye view image of the vehicle itself is the image information from the bird's-eye view perspective of the vehicle itself, and the side and rear steering image is the image information from the side and rear of the vehicle itself.

[0026] Furthermore, the display module is also used to obtain the first position of the object to be prompted ahead in the current frame; estimate the position of the object to be prompted ahead in at least one next frame in the world coordinate system based on the relative speed between the object to be prompted ahead and the vehicle, and obtain at least one second position; perform window averaging on at least one second position to obtain a third position; convert the third position to the vehicle coordinate system to obtain a fourth position; and display a prompt image of the object to be prompted or an image of a lane change warning light of the object to be prompted at the fourth position within the field of view of the augmented reality head-up display.

[0027] Furthermore, the augmented reality head-up display device of vehicle auxiliary element information also includes: a compensation module, which is used to determine first image information of at least one object to be prompted in front of the vehicle, a relative delay corresponding to the first image information, and a relative speed vector between at least one object to be prompted and the vehicle based on environmental perception data, wherein the first image information is image information of at least one object to be prompted in the vehicle coordinate system, and the relative delay includes a dynamic delay in the acquisition, transmission and calculation of the first image information, and a fixed delay in the rendering and display process; based on the relative speed vector and the relative delay, the first image information is delayed compensated to obtain second image information, so as to display the second image information at a corresponding position within the field of view of the augmented reality head-up display, wherein the second image information is image information of at least one object to be prompted in the vehicle coordinate system.

[0028] Furthermore, the compensation module is also used to determine first driving data of the vehicle and second driving data of at least one object to be prompted based on the environmental perception data, wherein the first driving data includes the current speed and heading angle of the vehicle in the vehicle coordinate system, and the second driving data includes the current speed and driving direction of at least one object to be prompted in the world coordinate system; convert the first driving data to the world coordinate system to obtain third driving data; determine a first relative velocity vector between the vehicle and at least one object to be prompted in the world coordinate system based on the second driving data and the third driving data; convert the first relative velocity vector to the vehicle coordinate system to obtain a relative velocity vector, wherein the relative velocity vector includes relative velocity information and relative direction information.

[0029] Furthermore, the compensation module is also used to determine the relative distance between the vehicle and at least one object to be prompted based on the relative velocity vector and the relative time delay; based on the relative distance and relative direction information, the first image information is delayed compensated in the vehicle coordinate system to obtain the second image information, wherein the time delay compensation is used to adjust the coordinates of the first image information to obtain the second image information.

[0030] Furthermore, the first determination module is also used to record the duration of the signal data being in the target state if the signal data is in the target state; in response to the duration being greater than a preset duration, determine that the vehicle is currently in a low visibility environment.

[0031] Furthermore, the first determination module is also used to determine that the signal data is in a target state in response to the wiper activation signal being in an activated state, the wiper operating speed being greater than a preset speed, the fog lamp activation signal being in an activated state, the rain light sensor data indicating that the rainfall is greater than a preset rainfall amount and the light intensity is greater than a preset light intensity.

[0032] Furthermore, the first acquisition module is also used to acquire first signal data collected by the vehicle through at least one sensor; and perform fusion processing on the first signal data to obtain signal data.

[0033] According to another aspect of an embodiment of the present invention, a vehicle is further provided, and the vehicle is used to execute the augmented reality head-up display method for implementing the vehicle auxiliary element information in various embodiments of the present invention.

[0034] According to another aspect of an embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the augmented reality head-up display method of vehicle auxiliary element information in various embodiments of the present invention.

[0035] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the augmented reality head-up display method of vehicle auxiliary element information in various embodiments of the present invention.

[0036] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the augmented reality head-up display method of vehicle auxiliary element information in each embodiment of the present invention is implemented.

[0037] In an embodiment of the present invention, first, signal data is collected from sensors carried by the vehicle, and based on these data, it is determined whether the vehicle is in a low visibility environment. If it is determined that the vehicle is in a low visibility environment, the current operating status of the augmented reality head-up display client will be automatically checked. If the augmented reality head-up display client is in a closed state, the vehicle will immediately start the augmented reality head-up display client and directly enter the augmented reality auxiliary element prompt mode to turn on targeted safety assisted driving functions. On the contrary, if the augmented reality head-up display is already running but in the conventional first display mode, the vehicle will quickly switch to the augmented reality auxiliary element prompt mode to adapt to the current severe weather conditions. When in the augmented reality auxiliary element prompt mode, the environmental perception data of the vehicle is further integrated and auxiliary element information is generated. Finally, the vehicle accurately displays the above-mentioned auxiliary element information, including the prompt image of the object to be prompted, at the most suitable position within the visible range of the augmented reality head-up display. By accurately identifying low-visibility environments, controlling the vehicle to enter the augmented reality auxiliary element prompt mode, and displaying the target auxiliary element information at the corresponding position of the vehicle, the purpose of significantly improving driving safety in low-visibility environments is achieved, thereby achieving a more intuitive, efficient and safe technical effect of presenting auxiliary element information, and thus solving the technical problem of blurred vision of the road ahead of the vehicle in low-visibility environments in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0039] Figure 1 It is a schematic diagram of a front vehicle and a lane line in a low visibility environment in the prior art;

[0040] Figure 2 is a schematic diagram of an augmented reality head-up display method for vehicle auxiliary element information according to an embodiment of the present invention;

[0041] Figure 3 is a schematic diagram of enhancing the display of a leading vehicle and lane lines in a low visibility environment according to an embodiment of the present invention;

[0042] Figure 4 is a schematic diagram of image information in an augmented reality head-up display mode according to an embodiment of the present invention;

[0043] Figure 5 is a schematic diagram of an augmented reality head-up display device for vehicle auxiliary element information according to an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0045] To help you better understand the content of this plan, here are some explanations of terms to help you understand:

[0046] Augmented Reality Head-Up Display (AR HUD): AR-HUD is a head-up display system that combines augmented reality technology. It can integrate real-time navigation, vehicle information, warning prompts, etc. with the external environment on the vehicle's windshield or transparent display and display them intuitively to the driver, thereby improving driving safety and convenience.

[0047] Kalman filter: Kalman filter is an effective algorithm for dynamic system state estimation, especially widely used in signal processing and control systems. Kalman filter can fuse the measurement data of multiple sensors and provide stable and accurate estimation results even when the data is not completely accurate or noisy.

[0048] Light Detection and Ranging (LiDAR): LiDAR is a sensor technology that uses pulsed lasers to measure distances. It can generate accurate three-dimensional environmental maps and is essential for autonomous vehicles, robot navigation, and the creation of high-precision maps. Especially in low-visibility environments, LiDAR can penetrate water vapor to a certain extent and provide relatively clear obstacle information.

[0049] Inertial Measurement Unit (IMU): IMU is a sensor combination including accelerometer and gyroscope, which is used to measure the acceleration, angular velocity and attitude of an object. For cars, IMU can monitor the vehicle's motion status in real time, such as acceleration, deceleration, turning, etc., which helps to enhance the dynamic calibration and stability of the reality display content.

[0050] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0051] According to an embodiment of the present invention, an embodiment of an augmented reality head-up display method for vehicle auxiliary element information is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0052] Figure 2 is a schematic diagram of an augmented reality head-up display method for vehicle auxiliary element information according to an embodiment of the present invention, such as Figure 2 As shown, the method comprises the following steps:

[0053] Step S20, obtaining signal data recognized by the vehicle through the sensor;

[0054] In the embodiment of the present invention, the vehicle can be understood as a vehicle, that is, the object to which the augmented reality head-up display method of vehicle auxiliary element information provided by the embodiment of the present invention is applied. For example, the vehicle can be equipped with an AR-HUD function, that is, equipped with an AR-HUD client, and can also be equipped with an intelligent driving assistance system, etc., which is not limited here.

[0055] Signal data can be understood as data generated by various sensors on the vehicle (such as cameras, radars, millimeter-wave radars, laser radars, wiper sensors, temperature and humidity sensors, etc.) when monitoring the surrounding environment and the vehicle status. For example, the signal data includes information about the external environment of the vehicle, such as weather conditions, road conditions, etc., and the signal data also includes information about the internal environment of the vehicle, such as the clarity or occlusion of the windshield of the vehicle, etc., which are not limited here.

[0056] Acquiring the signal data recognized by the vehicle through sensors can be understood as the process in which the vehicle's intelligent system actively collects and analyzes environmental information and vehicle status detected by various sensors on the vehicle. This process covers the conversion of the sensor's raw data into information that can be processed by the vehicle for subsequent decision-making and control.

[0057] In the embodiment of the present invention, real-time environmental signal data is collected by the vehicle's sensors for subsequent analysis to determine whether the vehicle is in a low-visibility environment, thereby ensuring that the vehicle can respond promptly based on the latest environmental changes and providing a data basis for the subsequent display of auxiliary elements.

[0058] Step S21, if it is determined that the signal data indicates that the vehicle is currently in a low visibility environment, then determining the current operating state of the augmented reality display client;

[0059] In the embodiment of the present invention, the low visibility environment can be understood as a situation where visibility is reduced due to external factors during driving, affecting the driver's line of sight. By way of example, it generally includes but is not limited to dense fog, heavy haze, heavy rain, blizzards, etc., which are not limited here.

[0060] The augmented reality display client can be integrated into the AR-HUD client in the vehicle. For example, AR-HUD is an advanced human-computer interaction technology that can project digital information on the front windshield of the vehicle, allowing the driver to see auxiliary information superimposed on the real environment while maintaining forward vision, such as navigation instructions, vehicle status, warning signals or other important data. By accurately positioning in the driver's natural field of vision, the driver's time to divert his or her gaze due to viewing the dashboard or central control screen is reduced, thereby improving driving safety and convenience. This is not limited here.

[0061] In an embodiment of the present invention, if the judgment signal data indicates that the vehicle is currently in a low visibility environment, then determining the current operating status of the augmented reality display client can be understood as, when the environmental signal data collected by the vehicle through various sensors indicates that the vehicle is traveling under extremely low visibility conditions, such as dense fog, rainy and snowy weather, etc., the vehicle will immediately check the current operating status of the AR-HUD client.

[0062] In an embodiment of the present invention, after determining that the vehicle is currently in a low visibility environment, the current operating status of the augmented reality display client carried by the vehicle will be immediately checked and confirmed to ensure that the AR-HUD client can respond to driving needs in a timely manner, thereby providing the driver with clear and intuitive driving assistance information when the line of sight is obstructed, thereby enhancing driving safety.

[0063] Step S22: if the augmented reality display client is in an off state, the augmented reality display client is turned on, and the augmented reality auxiliary element prompt mode is directly entered for display;

[0064] In the embodiment of the present invention, the augmented reality auxiliary element prompt mode can be understood as an operation mode in which the AR-HUD client provides specific visual auxiliary information to the driver in a low visibility environment. The augmented reality auxiliary element prompt mode can display auxiliary driving information in a preset display range of the vehicle (for example, on the front windshield), thereby improving the driving experience and driving safety.

[0065] If the augmented reality display client is in a closed state, the augmented reality display client is turned on, and the augmented reality auxiliary element prompt mode is directly entered for display. It can be understood that when the AR-HUD client is originally in a closed state, but the vehicle sensor detects that the visibility of the current environment is reduced, the present invention will automatically start the AR-HUD client, and immediately control the AR-HUD client to enter the augmented reality auxiliary element prompt mode and display it.

[0066] In an embodiment of the present invention, when it is determined that the vehicle is in a low visibility environment and the AR-HUD client is in a closed state, the AR-HUD client will be automatically activated, and the AR-HUD client will be immediately controlled to enter the augmented reality auxiliary element prompt mode and display it, which will be intuitively presented in the driver's field of vision through augmented reality technology, helping the driver to quickly and accurately understand the surrounding environment and the condition of the vehicle even when the line of sight is obstructed, so as to make timely and safe driving decisions, and significantly improve driving safety and efficiency in severe weather conditions.

[0067] Step S23, or, if the augmented reality display client is already running and in the first display mode, the first display mode is switched to the augmented reality auxiliary element prompt mode and displayed;

[0068] In the embodiment of the present invention, the first display mode state can be understood as the working mode or regular display mode that the augmented reality display client is currently in. Exemplarily, the first display mode state includes but is not limited to navigation mode, vehicle status mode, entertainment information mode, and ordinary augmented reality mode, which are not limited here.

[0069] If the augmented reality display client is already running and in the first display mode, the first display mode state is switched to the augmented reality auxiliary element prompt mode. It can be understood that when the vehicle sensor detects that the vehicle is in a low visibility environment and it is determined that the current first display mode state no longer adapts to the driving needs in this special environment, the present invention will automatically control the AR-HUD client to switch from the first display mode to the augmented reality auxiliary element prompt mode to cope with the driving challenges under low visibility conditions, improve the driver's perception of road conditions, and reduce the risk of traffic accidents caused by poor visibility, which is not limited here.

[0070] In an embodiment of the present invention, when the augmented reality display client is already in a running state and is currently in a first display mode state, the AR-HUD client will be immediately switched from the first display mode state to the augmented reality auxiliary element prompt mode to ensure that the driver can still obtain key road condition information when his vision is obstructed.

[0071] Step S24, obtaining the vehicle's environmental perception data in the augmented reality auxiliary element prompt mode;

[0072] In the embodiment of the present invention, the environmental perception data can be understood as the real-time information about the surrounding environment collected by the vehicle through a series of sensors. For example, the environmental perception data includes but is not limited to the front vehicle, lane lines, road edges, pedestrians, obstacles, and traffic signs captured by the front camera, which are not limited here.

[0073] In the augmented reality auxiliary element prompt mode, obtaining the vehicle's environmental perception data can be understood as, when the vehicle is in the augmented reality auxiliary element prompt mode, it will actively and preferentially call and process the real-time information collected by a series of vehicle-mounted sensors to support and optimize the assisted driving function provided by the AR-HUD client, which is not restricted here.

[0074] In an embodiment of the present invention, in the augmented reality auxiliary element prompt mode, a variety of sensors equipped in the vehicle (including but not limited to cameras, millimeter-wave radars, etc.) can collect key information in the surrounding environment, such as the position and movement trend of the vehicle in front, the state of the lane lines, the existence and position of potential obstacles, the visibility level of the driving environment, etc., thereby greatly improving driving safety and driving experience.

[0075] Step S25, determining auxiliary element information based on the environmental perception data;

[0076] In the embodiment of the present invention, auxiliary element information can be understood as information used to be displayed in a preset display range to assist safe driving. For example, the auxiliary element information includes, but is not limited to, information about the vehicle ahead, lane lines and road edges, obstacle and pedestrian warnings, road condition prompts, etc., which are not limited here.

[0077] Determining auxiliary element information based on environmental perception data can be understood as, when the augmented reality head-up display client is in the augmented reality auxiliary element prompt mode, it will use the collected environmental perception data to analyze, identify and judge key information that is helpful to the driver in the driving environment, which is not limited here.

[0078] In the embodiment of the present invention, based on the collected environmental perception data, the most valuable auxiliary element information for the driver is intelligently analyzed and extracted, thereby significantly improving the driver's safety perception and decision-making ability.

[0079] Step S26, displaying auxiliary element information at a corresponding position within the field of view of the augmented reality head-up display, wherein the auxiliary element information includes an object prompt image of the object to be prompted.

[0080] In the embodiment of the present invention, the object to be prompted can be understood as a static object or a dynamic object within the field of view of the AR-HUD client (i.e., the range that the driver can directly see through the AR-HUD client). Exemplarily, the dynamic object includes at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian or an animal within the field of view of the augmented reality head-up display of the vehicle, and the static object includes at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian or an animal within the field of view of the augmented reality head-up display of the vehicle, which is not limited here.

[0081] The prompt image of the object to be prompted can be understood as the image information of a static object or a dynamic object displayed within the field of view of the AR-HUD client. Exemplarily, the prompt image of a static object includes images of a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, and a non-moving pedestrian or animal displayed within the field of view of the vehicle's AR-HUD, which are not limited here. The prompt image of a dynamic object includes images of a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian or an animal displayed within the field of view of the vehicle's AR-HUD, which are not limited here.

[0082] Displaying auxiliary element information at the corresponding position within the field of view of the augmented reality head-up display can be understood as the AR-HUD client superimposing auxiliary element information (such as road signs, obstacle warnings, navigation instructions, etc.) in the form of augmented reality on the actual environment of the driver's front line of sight based on the real-time data obtained by the vehicle's sensors and cameras. It is understandable that the prompt image can be in the outline of the vehicle image or next to the vehicle outline, which is not limited here. Therefore, different auxiliary element information corresponds to different display positions, and the display position also changes in real time, that is, the position of the information display is consistent with the position observed by the driver in the real world. For example, the prompt image of the vehicle in front appears directly at its actual position, without the driver shifting his line of sight to identify the information, thereby improving driving safety and ease of operation, which is not limited here.

[0083] In the embodiment of the present invention, the analyzed and processed auxiliary element information, especially the prompt image of the object to be prompted in front, is accurately presented at the corresponding position within the field of view of the AR-HUD client. By converting the real-time status of the object to be prompted, such as position, speed and direction, into intuitive visual prompts and seamlessly integrating them with the driver's line of sight, the driver's safety awareness and reaction ability in low visibility conditions are greatly enhanced.

[0084] It is understandable that although both using the car screen to display the navigation information during vehicle driving and displaying the navigation information during vehicle driving through AR-HUD can provide navigation guidance, displaying navigation information through AR-HUD does not use the same navigation information in place of a different display subject. Instead, there are essential differences in the generation and implementation logic and effect of the displayed navigation information. The difference between the two determines their respective characteristics and application scenarios.

[0085] The core of AR-HUD is to integrate the navigation guidance light carpet with the actual road, and integrate the perceived lane lines ahead, motor vehicles, non-motor vehicles, pedestrians or animals in the surrounding environment and other information that needs to be prompted with the real environment, and present it on the windshield in front of the driver's line of sight. In terms of effect, this alignment of navigation virtual information with the real world keeps the virtual information highly consistent with the real world elements, providing users with an intuitive and immersive driving assistance experience.

[0086] In contrast, the navigation information on the car screen is presented. The navigation route (not the navigation guidance light carpet), the car and surrounding vehicles and other virtual information are rendered on the fixed display screen in the car through animation. The navigation route and virtual information do not need to be spatially integrated with the external environment. The main display function is to show the user what the surrounding environment is like, and there is no such high demand for accuracy.

[0087] The specific differences are as follows:

[0088] Table 1

[0089]

[0090]

[0091] As shown in Table 1, the essential difference between AR-HUD and traditional car screens in navigation display functions is that AR-HUD can directly overlay navigation information within the driver's field of vision due to its augmented reality characteristics, and can display navigation guidance light carpets on the road where the vehicle is currently located, so that the navigation guidance light carpets and the actual road are aligned. Users can clearly know how to drive on the road ahead, which is very intuitive. However, the navigation on the car screen does not require direct visual integration with the road, and does not display navigation information through light carpets. Instead, the accessible roads in front of the vehicle are marked with colors to indicate to users which roads are accessible. Users know which roads can be taken, and as for which road to take, users decide for themselves. In addition, AR-HUD can display information such as the lane lines ahead within a certain range ahead, motor vehicles, non-motor vehicles, pedestrians or animals in the surrounding environment, and intuitively prompt users of external factors that may affect driving operations. Especially in environments with low visibility, users may not notice what the surrounding environment is like. This information can greatly help users make decisions to avoid traffic accidents. However, the navigation on the car screen will render some surrounding objects, but users can only roughly understand that there may be an object around, and users cannot intuitively feel the specific location of the object.

[0092] Regarding whether it is necessary to predict the position of the vehicle in front in real time, since the underlying implementation logic of these two technologies is different, for example, assuming that in low visibility, the position element of the vehicle in front is displayed through AR-HUD to help users identify the distance between the vehicle and the vehicle in front, how to ensure that this position element fits the vehicle in front is very important for users. The position of the vehicle in front needs to be predicted to be more consistent. For example, if the vehicle in front suddenly accelerates or decelerates, in order to make the animation rendering on the AR-HUD keep up with the vehicle in front, the position of the vehicle in front needs to be predicted. Specifically, it is necessary to obtain multiple predicted positions of the vehicle in front (for example, the predicted position of some vehicles in front is the position after acceleration, and the predicted position of some vehicles in front is the position after deceleration), and then the position of the vehicle in front in the current frame is averaged by window to obtain a predicted position. At this time, this predicted position can be displayed on the AR-HUD. At this time, even if the vehicle in front suddenly accelerates, the predicted position is relatively consistent.

[0093] The navigation screen displayed on the car screen will have an animation rendering of the vehicle in front, but there is no strong requirement for this animation rendering to be able to fit the vehicle in front. It does not matter even if there is a certain distance difference between the rendered position of the vehicle in front and the actual position. The car screen only needs to indicate an approximate relative position of the vehicle in front, and a particularly accurate position of the vehicle in front is not required. In other words, the navigation displayed on the car screen does not predict the speed of the vehicle in front, but detects and calculates in real time through perception data whether the vehicle in front will collide with the vehicle in front at present, without knowing whether the vehicle in front will collide with the vehicle in front in the future.

[0094] Regarding whether there is a problem of navigation guidance light blanket exceeding the frame, since the display field of view of AR-HUD is only the content within a certain range in front of the vehicle, and the navigation guidance light blanket must be consistent with the road conditions. For example, if there is a right turn or a U-turn ahead, in this case, the navigation guidance light blanket will exceed the frame. However, the picture displayed on the car screen includes the passable road ahead (and the navigation guidance light blanket is not displayed). Even if there is a right turn or a U-turn ahead, the passable road ahead will be displayed in the picture, so the car screen does not have a problem of navigation guidance light blanket exceeding the frame.

[0095] The method proposed in the embodiment of the present application is applicable to user manual driving mode and navigation through AR-HUD, and is also applicable to automatic driving mode and navigation through AR-HUD.

[0096] In an embodiment of the present invention, signal data is first collected from the sensors carried by the vehicle, and based on these data, it is determined whether the vehicle is in a low visibility environment. If it is determined that the vehicle is in a low visibility environment, the current operating status of the augmented reality head-up display client will be automatically checked. If the augmented reality head-up display AR-HUD client is in a closed state, the vehicle will immediately start the AR-HUD client and directly enter the augmented reality auxiliary element prompt mode to turn on the targeted safety assisted driving function. On the contrary, if the augmented reality head-up display is already running but in the conventional first display mode, the vehicle will quickly switch to the augmented reality auxiliary element prompt mode to adapt to the current severe weather conditions. When in the augmented reality auxiliary element prompt mode, the environmental perception data of the vehicle is further integrated and auxiliary element information is generated. Finally, the vehicle accurately displays the above-mentioned auxiliary element information, including the prompt image of the object to be prompted, at the most suitable position within the visible range of the augmented reality head-up display. By accurately identifying low-visibility environments, controlling the vehicle to enter the augmented reality auxiliary element prompt mode, and displaying the target auxiliary element information at the corresponding position of the vehicle, the purpose of significantly improving driving safety in low-visibility environments is achieved, thereby achieving a more intuitive, efficient and safe technical effect of presenting auxiliary element information, and thus solving the technical problem of blurred vision of the road ahead of the vehicle in low-visibility environments in related technologies.

[0097] Optionally, in step S25, the auxiliary element information also includes a lane line image of the current lane where the vehicle is located, and determining the auxiliary element information based on the environmental perception data includes the following steps:

[0098] Step S251, acquiring a lane line image of the current lane based on the environmental perception data, and rendering the lane line image of the current lane at a corresponding position within the field of view of the augmented reality head-up display.

[0099] In the embodiment of the present invention, it can be understood that the present invention can determine the auxiliary element information based on the environmental perception data in three driving modes (i.e., pure manual mode, manual + navigation mode, and pure self-driving mode). In the pure manual mode, the vehicle is completely controlled by the driver. The auxiliary element information in the pure manual mode includes the vehicle in front, the lane line and the road boundary, obstacles on the road surface (such as pedestrians and cyclists), etc., which are not limited here. In the manual + navigation mode, the driver is responsible for the basic operation of the vehicle, but the vehicle is equipped with a navigation system to provide route guidance and auxiliary information. The auxiliary element information in the manual + navigation mode includes navigation instructions (such as turning, changing lanes), road condition information, etc., which are not limited here. In the pure self-driving mode, the vehicle is in a fully automatic driving state, and the driver does not need to perform any physical operation, but only needs to monitor the vehicle operation status. The auxiliary element information in the pure self-driving mode includes the automatic driving state, decision path, environmental perception feedback (such as the real-time location of surrounding vehicles and pedestrians), etc., which are not limited here. Among them, in the scenario where the user is driving purely manually, the auxiliary element information does not include the prompt information of changing lanes, while the user in the manual + navigation mode and the pure self-driving mode, the auxiliary element information includes the prompt information of changing lanes.

[0100] The lane line image of the current lane where the vehicle is located can be understood as a virtual image representing the actual road lane line that is accurately displayed in the driver's field of view through AR-HUD technology. For example, the lane line image of the current lane where the vehicle is located includes the position, shape (solid line or dashed line), direction and possible offset of the lane line, which is not limited here.

[0101] Based on the environmental perception data, the lane line image of the current lane is obtained, and the lane line image of the current lane is rendered at the corresponding position within the field of view of the augmented reality head-up display. It can be understood that the vehicle identifies and constructs a virtual lane line image through deep learning and image processing of the collected data. The lane line image highly restores the position, shape and direction of the actual lane line. Subsequently, these lane line images are superimposed on the field of view of the AR-HUD client through augmented reality technology, and aligned with the real lane line in the driver's line of sight, ensuring that the driver does not need to be distracted to check the instrument or the central control screen, and can clearly see the lane line status directly in front of the line of sight, which significantly improves driving safety and driving experience.

[0102] In an embodiment of the present invention, the vehicle's environmental perception system is used to capture current lane line information, generate a virtual lane line image, and accurately project it at a corresponding position in the driver's field of view through AR-HUD technology to enhance the visibility of lane lines in adverse weather conditions and improve driving safety.

[0103] For example, Figure 3 is a schematic diagram of enhancing the display of the preceding vehicle and lane lines in a low visibility environment according to an embodiment of the present invention, such as Figure 3 As shown, in the augmented reality head-up display mode, the display content will be adjusted according to the visibility environment to highlight important information related to driving safety, such as the precise position of the object to be prompted in front (such as the rear position of the vehicle in front) and a clear lane line image of the current lane, thereby improving driving safety and efficiency. There is no restriction here.

[0104] Optionally, in step S25, the auxiliary element information further includes a lane line image of the lane to be driven, the lane line image of the lane to be driven includes a lane line image of the current lane where the vehicle is located and a lane line image of the lane to be changed to, and determining the auxiliary element information based on the environmental perception data further includes the following steps:

[0105] Step S252, obtaining lane-level light blanket control line data in a map coordinate system from a cloud server;

[0106] Step S253, determining the lane to be driven by the vehicle based on the lane-level light blanket control line data, wherein the lane to be driven includes the current lane where the vehicle is located and the lane to be changed by the vehicle;

[0107] Step S254, determining first lane line element information corresponding to the lane to be driven from the environmental perception data;

[0108] Step S255, converting the first lane line element information from the world coordinate system to the vehicle coordinate system to obtain the second lane line element information in the vehicle coordinate system;

[0109] Step S256, fusing the second lane line element information and the lane-level light blanket control line data to obtain the third lane line element information in the vehicle coordinate system;

[0110] Step S257, determining the lane line image of the lane to be driven based on the third lane line element information.

[0111] In the embodiment of the present invention, the cloud server can be understood as a remote data processing center that can store and provide a large amount of map data, traffic information, road features, etc. For example, in the field of intelligent driving, the cloud server is usually used to update and maintain high-precision maps, and provide lane-level detailed information related to the current position of the vehicle, which is not limited here.

[0112] The map coordinate system can be understood as a standard reference system used to determine geographic location, usually based on the longitude and latitude coordinates of the earth's surface. For example, the data in the map coordinate system provides the specific location and shape of elements such as roads, lanes, and intersections, which is the basis of high-precision map information and is not limited here.

[0113] The lane-level light carpet control line data can be understood as a predefined lane-level navigation information obtained from the cloud server and used to render the light carpet.

[0114] The first lane line element information can be understood as lane line features extracted from environmental perception data (such as information captured by sensors such as cameras and radars). For example, the first lane line element information includes the position, shape, direction, etc. of the lane line, reflecting the actual lane line state of the road the driver is currently on, which is not limited here.

[0115] The vehicle coordinate system can be understood as a coordinate system established with the vehicle itself as the origin, which is used to describe the relative position of the vehicle's surrounding environment. For example, the data in the vehicle coordinate system is easy to understand and process by the vehicle's internal system, which is the key to achieving AR-HUD lane line image fitting, and is not limited here.

[0116] The second lane line element information can be understood as the information obtained by converting the first lane line element information from the world coordinate system to the vehicle coordinate system. For example, the purpose of the conversion is to ensure that the lane line image can be accurately fitted according to the motion state and position of the vehicle when displayed in the AR-HUD client, which is not limited here.

[0117] The third lane line element information can be understood as the comprehensive information obtained by fusing the lane-level light blanket control line data with the second lane line element information. For example, through fusion, the insufficiency of environmental perception data under certain conditions can be compensated. For example, the recognition of lane lines may be limited in rainy and foggy weather. The fused data can provide more comprehensive and reliable lane line information, which is not limited here.

[0118] It is understandable that, considering that objects at different distances may have differences in perception accuracy and the validity of data sources, AR-HUD adopts a flexible data fusion strategy to optimize the authenticity and reliability of information, that is, the information fusion of close objects relies more on real-time perception data, while the information fusion of distant objects makes more use of map data and other pre-acquired information. For example, for dynamic objects such as vehicles in front of the vehicle, pedestrians, bicycles, and static objects on the road such as traffic signs, the AR-HUD system tends to use high-precision real-time perception data, such as camera images, millimeter-wave radar signals, LiDAR point cloud data, etc. In this case, the weight of perception data is greater than that of map data to ensure the immediacy and accuracy of information. On the contrary, for objects far away from the vehicle, such as distant traffic lights, building outlines, road planning, etc., the system relies more on map data and historical information. In long-distance observation, perception data may be interfered by factors such as weather conditions and light effects, resulting in a decrease in information quality. Map data provides more stable and comprehensive background information, including road structure, fixed landmarks, predicted traffic conditions, etc., and the above information has a high degree of credibility over a large range and long distance. Therefore, for distant objects, the weight of map data is greater than that of perception data to build a stable long-distance view, which is not restricted here.

[0119] Acquiring lane-level light blanket control line data in the map coordinate system from the cloud server can be understood as the vehicle receiving data containing high-precision lane information through communication with the cloud. The above information defines the precise boundaries and other key features of the lane in the map coordinate system for navigation and assisted driving, which is not restricted here.

[0120] Determining the lane to be driven by the ego vehicle based on the lane-level light blanket control line data can be understood as the ego vehicle analyzing the lane-level navigation data obtained from the cloud, combining the ego vehicle's current position and motion status, and intelligently identifying the ego vehicle's current lane and possible target lanes for lane change, providing a reference for subsequent lane line element information fusion and image generation, which is not restricted here.

[0121] Determining the first lane line element information corresponding to the lane to be traveled from the environmental perception data can be understood as using the sensors on the vehicle (such as cameras, radars, etc.) to detect and analyze the lane lines in the environment around the vehicle in real time, including details such as their position, shape, and direction, to obtain real-time lane line data related to the lane to be traveled, which is not limited here.

[0122] Converting the first lane line element information from the world coordinate system to the ego-vehicle coordinate system to obtain the second lane line element information in the ego-vehicle coordinate system can be understood as converting the real-time detected lane line information (first lane line element information) from the universal world coordinate system to the ego-vehicle-centered ego-vehicle coordinate system, ensuring that the lane line data matches the real-time position and motion state of the ego-vehicle, and providing accurate positioning information for the lane line image rendering of the AR-HUD, which is not restricted here.

[0123] The fusion of the second lane line element information and the lane-level light blanket control line data to obtain the third lane line element information in the vehicle coordinate system can be understood as combining the converted real-time lane line data (second lane line element information) with the predefined lane-level light blanket control line data, and processing them through a data fusion algorithm to generate more comprehensive and accurate lane line information (third lane line element information) to cope with the uncertainty of environmental perception data, which is not restricted here.

[0124] Determining the lane line image of the lane to be traveled based on the third lane line element information can be understood as using the fused lane line information (third lane line element information) to generate and display a lane line image that accurately matches the lane to be traveled through AR-HUD technology, providing the driver with intuitive lane navigation assistance, especially in low-visibility weather conditions, to enhance driving safety and visualization effects, which is not restricted here.

[0125] In an embodiment of the present invention, light carpet control line data containing lane-level details is obtained from a cloud server to provide high-precision lane information for the vehicle. Then, the current and upcoming lanes of the vehicle are determined based on these data. Subsequently, lane line feature information related to the lane to be driven is extracted from the environmental perception data, and the coordinate system of this information is converted to match the perspective of the vehicle. Based on the converted information, it is fused with the lane-level light carpet control line data to generate more accurate lane line element information. Finally, the lane line image of the lane to be driven is generated using the fused information, and displayed in the field of view directly in front of the driver through AR-HUD technology. Therefore, the AR-HUD client can integrate the vehicle's environmental perception and cloud data to achieve accurate identification and enhanced display of lane lines, provide key driving assistance information to the driver, especially in environments with obstructed vision, and improve driving safety and comfort.

[0126] Optionally, in step S25, the auxiliary element information further includes a lane line image of the lane to be driven, the lane line image of the lane to be driven includes a lane line image of the current lane where the vehicle is located and a lane line image of the lane to be changed to, and determining the auxiliary element information based on the environmental perception data further includes the following steps:

[0127] Calculate the lane-level light blanket control line data in the vehicle coordinate system based on the navigation direction data and the environment perception data;

[0128] Determine the lane to be driven by the vehicle based on the lane-level light blanket control line data, wherein the lane to be driven includes the current lane where the vehicle is located and the lane to be changed by the vehicle;

[0129] Determine lane line element information corresponding to the lane to be driven from the environmental perception data;

[0130] The lane line image of the lane to be driven is determined based on the lane line element information.

[0131] In the embodiment of the present invention, the navigation direction data can be understood as the driving direction information provided by the vehicle navigation system based on the accurate map. For example, the navigation direction data can be based on the current position of the vehicle, the destination and the route planning, indicating the lane and lane change direction that the vehicle should travel, which is not limited here.

[0132] Lane line element information can be understood as detailed information about the lane line extracted from the vehicle's environmental perception data, including but not limited to the lane line's position, shape, width, color, and its orientation and distance relative to the vehicle, etc., which is not limited here.

[0133] Calculating the lane-level light blanket control line data in the ego-vehicle coordinate system based on the navigation direction data and the environmental perception data can be understood as combining the ego-vehicle's navigation information (such as the destination, recommended route, etc.) and the road conditions captured by the environmental perception sensors (such as cameras, radars), and calculating the lane boundary data with the ego-vehicle as the reference system through a series of data processing and coordinate conversion algorithms, that is, the expression form of the lane-level light blanket control line data in the ego-vehicle coordinate system, which is not restricted here.

[0134] Determining the lane in which the vehicle is to travel based on the lane-level light carpet control line data can be understood as the vehicle analyzing and calculating the lane-level light carpet control line data to identify the lane in which the vehicle is currently located and its potential target lane for lane change, so as to provide the driver with important lane information required for the upcoming driving or lane change, which is not limited here.

[0135] Determining the lane line element information corresponding to the lane to be driven from the environmental perception data can be understood as using the vehicle's environmental perception equipment, such as cameras and lidar, to detect and analyze in real time the lane line features related to the lane to be driven in the vehicle's surrounding environment, including the position, shape, direction, etc. of the lane line, to provide real-time basis for the AR-HUD lane line enhanced display, which is not restricted here.

[0136] Determining the lane line image of the lane to be driven based on the lane line element information can be understood as using the analyzed lane line element information in combination with AR-HUD technology to generate a virtual image of the lane line that completely matches the lane to be driven, and accurately superimposing these images on the actual line of sight, especially in low-visibility weather conditions, to provide the driver with intuitive lane navigation assistance, thereby improving driving safety and accuracy, which is not restricted here.

[0137] In an embodiment of the present invention, first, the vehicle calculates the lane-level light blanket control line data based on the navigation direction data and the environmental perception data, and determines the lane where the vehicle is currently located and the target lane to be changed based on the lane-level light blanket control line data. Then, the environmental perception sensors (such as cameras and radars) on the vehicle are used to detect the lane line features related to the lane to be driven, such as position, shape and direction, in real time to form lane line element information. Finally, based on these lane line element information, combined with AR-HUD technology, a lane line enhanced image that accurately matches the lane to be driven is generated in the driver's field of view directly in front of the driver, providing the driver with a clear lane boundary reference. Through the above steps, the entire AR-HUD client effectively integrates navigation, environmental perception and display technologies, and provides drivers with accurate lane line guidance under complex driving conditions. It is a key technical link for the smart car AR-HUD client to improve driving experience and safety.

[0138] Optionally, the signal data includes external sensor data of the vehicle and internal sensor data of the vehicle, and the augmented reality head-up display method of vehicle auxiliary element information further includes the following steps:

[0139] If the vehicle's external sensor data indicates that the vehicle is currently in at least one of a heavy rain environment, a smog environment, a haze environment, a snowstorm environment, a sandstorm environment, and a strong light environment, then it is determined that the vehicle is currently in a low visibility environment;

[0140] If the internal sensor data of the vehicle indicates that at least one of the front windshield window of the vehicle is fogged or the front windshield is suddenly blocked, it is determined that the vehicle is currently in a low visibility environment.

[0141] In the embodiment of the present invention, external sensor data can be understood as information provided by sensors equipped on the vehicle for monitoring the state of the surrounding environment. Exemplarily, external sensors mainly include but are not limited to rain sensors, air quality sensors, optical sensors, image sensors (cameras), etc. The external sensor data together constitute the vehicle's perception of the state of the surrounding environment. When any one or more indicators reach the set threshold, it will be determined that the vehicle is in a low visibility environment and trigger corresponding auxiliary driving measures, which are not limited here.

[0142] Internal sensor data can be understood as information used to reflect the in-vehicle environment. Exemplarily, the internal sensors include, but are not limited to, humidity sensors and occlusion detection sensors, which are not limited here.

[0143] If the external sensor data of the vehicle indicates that the vehicle is currently in at least one of the following environments: heavy rain, smoke, haze, blizzard, sandstorm, and strong light, then determining that the vehicle is currently in a low visibility environment can be understood as when the signal detected by the environmental perception sensor outside the vehicle (such as a rain sensor, optical sensor, image sensor, etc.) indicates the existence of the above-mentioned severe weather or environmental conditions, it will automatically recognize that the vehicle is in a low visibility environment, which is not restricted here.

[0144] If the internal sensor data of the vehicle indicates that the front windshield of the vehicle is fogged or the front windshield is suddenly blocked, then the determination that the vehicle is currently in a low visibility environment can be understood as the data monitored by the on-board internal sensors (such as humidity sensors, occlusion detection sensors) showing that the internal environment of the vehicle affects the driver's line of sight, such as fogging inside the front windshield or being blocked by objects outside, resulting in obstruction of direct vision. In the above case, even if the outside weather is good, driving safety will be affected, so it will also be identified as a low visibility environment, triggering the corresponding auxiliary function to ensure that the driver can obtain clear field of view information and improve driving safety, which is not restricted here.

[0145] In the embodiment of the present invention, through the comprehensive monitoring of the interior and external environment of the vehicle by external sensors and internal sensors, the impact of the external and internal environment of the vehicle on the driver's vision can be comprehensively evaluated. Once a low visibility environment is confirmed, driving assistance functions including AR-HUD will be activated to provide additional visual information and guidance to help the driver drive safely under poor visibility conditions.

[0146] Optionally, the auxiliary element information also includes at least one of the following: a road edge line image, an image of a lane change warning light of an object to be prompted, a bird's-eye view image of the vehicle itself, and a side and rear steering image, wherein the lane change warning image of the object to be prompted is the image information of the lane change warning light of the vehicle in front, the bird's-eye view image of the vehicle itself is the image information from the bird's-eye view of the vehicle itself, and the side and rear steering image is the image information from the side and rear of the vehicle itself.

[0147] In the embodiment of the present invention, the road edge line image can be understood as a virtual road edge line generated by the AR-HUD client within the driver's line of sight, which detects and identifies road edge information in real time based on environmental perception data (such as cameras, radars), and converts this information into an image, which is superimposed on the actual driving field of view, and is not limited here.

[0148] The lane change warning light image of the object to be prompted can be understood as AR-HUD technology presenting the lane change intention or status of surrounding vehicles in the form of virtual warning lights in the driver's field of vision. For example, when it is detected that the vehicle ahead is preparing to or is changing lanes, the AR-HUD client can generate a lane change warning light image in real time to remind the driver to pay attention to the traffic dynamics ahead and respond in time to avoid dangerous situations. This is not limited here.

[0149] The bird's-eye view image of the vehicle can be understood as the image of the vehicle's surrounding environment from a bird's-eye view provided by the AR-HUD client, which simulates the perspective of looking down from the top of the vehicle and displays the traffic conditions and obstacles around the vehicle. Exemplarily, the AR-HUD client constructs a three-dimensional model of the vehicle's surrounding environment by fusing multi-sensor data (such as cameras, radars, etc.), and generates real-time image information from a bird's-eye view similar to the perspective from above the vehicle to display to the driver. The image of the vehicle from a bird's-eye view can provide a more comprehensive perspective of the vehicle's surrounding environment, especially in complex traffic conditions, such as narrow urban streets, parking lots or intersections, to help the driver better judge the relative position of the vehicle and obstacles and make safer driving decisions, which is not restricted here.

[0150] The side and rear steering images can be understood as real-time images (including left and right rear images, i.e. corresponding to the left and right rearview mirrors) captured by the cameras on the side and rear of the vehicle when the vehicle is changing lanes. For example, when the driver is about to change lanes, the vehicle will automatically activate the camera on the side and rear of the vehicle to capture real-time images of the area, and present these images in the driver's line of sight in front of him through AR-HUD technology to enhance the driver's perception of the traffic conditions on the side and rear. Especially in low visibility environments such as rain, fog, and at night, the side and rear steering images make up for the lack of field of view of the exterior rearview mirrors, improve safety when changing lanes, and reduce accidents caused by blind spots, which are not restricted here.

[0151] The lane change warning image of the object to be prompted is the image information of the lane change warning light of the vehicle in front. It can be understood that when the environmental perception system of the vehicle detects that other vehicles in front are preparing or changing lanes, the AR-HUD client will generate a virtual image to simulate the position and status of the lane change indicator light (such as a turn signal) of the vehicle in front in the driver's line of sight. The lane change warning image of the object to be prompted not only shows the intention of the lane changing vehicle, but also takes into account the relative position and dynamics of the vehicle, helping the driver to respond quickly and avoid potential safety risks, which is not restricted here.

[0152] In the embodiment of the present invention, the provision of the above-mentioned auxiliary element information utilizes advanced sensing technology and augmented reality display technology, aiming to create a more intuitive and safer driving environment, especially in severe weather or complex road conditions, to improve the driver's decision-making speed and accuracy.

[0153] For example, Figure 4 is a schematic diagram of image information in an augmented reality head-up display mode according to an embodiment of the present invention. Figure 4 As shown, in the augmented reality head-up display mode of the embodiment of the present invention, a clear lane line image of the current lane, the precise position of the object to be prompted in front (i.e., a clear rear end of the vehicle in front), real-time images of the side and rear of the vehicle (i.e., side and rear steering image information), road information of the current road (such as road speed limit, road warning information, etc.), driving information of the vehicle (such as the current driving speed of the vehicle, etc.), and navigation information (such as the planned path, recommended driving speed, etc.) can be displayed within the field of view of the augmented reality head-up display. By viewing the image information within the field of view of the augmented reality head-up display, the driver can be helped to predict and understand the traffic conditions around the vehicle in a timely manner, so that the driver can react in advance to avoid dangerous driving situations caused by driving out of the road boundary or running over the lane line of the lane where the vehicle is located, following the vehicle too close, the rear vehicle suddenly accelerating or changing lanes, the vehicle speeding too fast or deviating from the navigation route, etc., which are not limited here.

[0154] Optionally, in step S26, displaying the auxiliary element information at a corresponding position within the augmented reality head-up display field of view includes the following steps:

[0155] Step S261, obtaining the first position of the object to be prompted in the current frame;

[0156] Step S262, estimating the position of the object to be prompted in front in at least one next frame in the world coordinate system based on the relative speed between the object to be prompted in front and the vehicle, to obtain at least one second position;

[0157] Step S263, performing window averaging on at least one second position to obtain a third position;

[0158] Step S264, converting the third position to the vehicle coordinate system to obtain a fourth position;

[0159] Step S265: displaying a prompt image of the object to be prompted or a lane change warning light image of the object to be prompted at a fourth position within the field of view of the augmented reality head-up display.

[0160] In the embodiment of the present invention, the first position can be understood as the actual position of the object to be prompted in the front in the world coordinate system in the current frame (i.e., the current time point). For example, the first position is detected and calculated in real time by a sensor (such as a camera, radar, LiDAR) on the vehicle, which is not limited here.

[0161] The second position can be understood as that after obtaining the first position, the vehicle will use the relative speed information of the object to be prompted in front and the vehicle to estimate the possible position of the object to be prompted in at least one next frame (i.e., the next time point or multiple future time points) in the world coordinate system, which is not limited here.

[0162] Window averaging can be understood as the process of performing window averaging on the estimated second position of the vehicle in order to improve the accuracy of the estimated position and avoid display instability caused by sensor data fluctuations or algorithm prediction errors. For example, the estimated position of the preceding vehicle for 50 frames is window averaged, and old data is removed and new data is continuously added, which is not limited here.

[0163] The third position can be understood as the average predicted position of the front object to be prompted in the next frame or the next few frames obtained after window averaging processing, which is not limited here.

[0164] The fourth position may be understood as a position obtained by converting the third position from the world coordinate system to the vehicle coordinate system, which is not limited here.

[0165] Obtaining the first position of the object to be prompted ahead in the current frame can be understood as using the sensors (such as cameras, radars) equipped on the vehicle to capture the real-time position information of the object to be prompted ahead (such as other vehicles, pedestrians) at the current time point. This information is expressed in the world coordinate system, providing an instantaneous snapshot of the object's position relative to the current position of the vehicle, which is not restricted here.

[0166] Based on the relative speed between the object to be prompted in front and the vehicle, the position of the object to be prompted in front in at least one next frame is estimated in the world coordinate system to obtain at least one second position. It can be understood that based on the position of the object to be prompted in front detected in the current frame and its relative speed with the vehicle, the physical kinematics principle and the prediction algorithm are used to estimate the possible position of the object to be prompted in the next one or more frames. The second position is a set of predicted positions of objects to be prompted, taking into account the possible movement path and speed changes of the object, which are not limited here.

[0167] Window averaging of at least one second position to obtain the third position can be understood as, in order to improve the stability of the estimated position and avoid fluctuations in sensor data or instantaneous errors in algorithm prediction, sliding window averaging of multiple second positions (i.e., multiple predicted positions) is performed, that is, the average value of the predicted positions within a certain time window is taken to obtain a smoother and more reliable third position as the future average predicted position of the object to be prompted ahead, which is not restricted here.

[0168] Converting the third position to the vehicle coordinate system to obtain the fourth position can be understood as converting the third position (i.e., the average predicted position of the object to be prompted in front) from the world coordinate system to the vehicle coordinate system centered on the vehicle, so as to accurately display it on the AR-HUD. The fourth position directly reflects the relative position of the object to be prompted and the vehicle, ensuring that the AR display content is fully aligned with the driver's line of sight, and improving the effect of augmented reality, which is not limited here.

[0169] Displaying the prompt image of the object to be prompted or the lane change prompt light image of the object to be prompted at the fourth position within the field of view of the augmented reality head-up display can be understood as using AR-HUD technology to accurately fit the prompt image of the object to be prompted at the fourth position (such as the vehicle outline, distance mark) or its lane change intention (such as turn signal) in the form of a virtual image within the driver's field of view, that is, display it within the field of view of the AR-HUD client. This display method not only intuitively informs the driver of the position and behavior of the object to be prompted in front, but also takes into account the driver's visual habits, reduces interference with driving safety, and improves the driver's reaction speed and safety, which is not restricted here.

[0170] In an embodiment of the present invention, the current position of the object to be prompted in front is captured in real time by the sensors of the vehicle (such as cameras, radars), and this position is the coordinate in the world coordinate system. Based on the relative speed between the object to be prompted and the vehicle, the possible position of the object to be prompted at one or more time points in the future is estimated in the world coordinate system to obtain one or more second positions. In order to improve the stability and accuracy of the prediction, at least one second position is subjected to window averaging processing, that is, the average value of the predicted position within a certain time window is taken to obtain a smoother and more reliable third position. The third position is converted from the world coordinate system to the vehicle coordinate system, and the predicted position of the object to be prompted is expressed in coordinates relative to the vehicle to obtain a fourth position. Finally, within the field of view of the AR-HUD client, the prompt image of the object to be prompted or the lane change warning light image thereof is displayed in real time according to the fourth position. These images are precisely aligned with the driver's line of sight, providing intuitive visual assistance to help the driver better judge the traffic conditions ahead.

[0171] Optionally, the augmented reality head-up display method of vehicle auxiliary element information further includes the following steps:

[0172] Determine, based on the environmental perception data, first image information of at least one object to be prompted in front of the vehicle, a relative delay corresponding to the first image information, and a relative speed vector between at least one object to be prompted and the vehicle, wherein the first image information is image information of at least one object to be prompted in the vehicle coordinate system, and the relative delay includes a dynamic delay of the first image information in the process of acquisition, transmission and calculation, and a fixed delay in the process of rendering and display;

[0173] The first image information is delayed compensated based on the relative velocity vector and the relative delay to obtain the second image information, so as to display the second image information at a corresponding position within the field of view of the augmented reality head-up display, wherein the second image information is image information of at least one object to be prompted in the vehicle coordinate system.

[0174] In an embodiment of the present invention, the first image information can be understood as real-time image information of at least one object to be prompted in front of the vehicle in the vehicle coordinate system collected by the vehicle's environmental perception system (including but not limited to cameras, radars, etc.) at a certain point in time, reflecting the position, size and appearance characteristics of the object to be prompted relative to the vehicle at the current moment, which is not limited here.

[0175] The relative delay corresponding to the first image information can be understood as the delay from the collection of sensor data to processing and then to the final display on the AR-HUD client in an actual system, which is not limited here.

[0176] The second image information can be understood as calculating the actual position of the object to be prompted at the display time based on the relative speed vector between the object to be prompted and the vehicle and the relative time delay corresponding to the first image information, and generating corresponding image information, i.e., the second image information, which is not limited here.

[0177] Determining the first image information of at least one object to be prompted in front of the vehicle, the relative delay corresponding to the first image information, and the relative speed vector between at least one object to be prompted and the vehicle based on the environmental perception data can be understood as using the environmental data collected by sensors (such as cameras, radars, etc.) equipped on the vehicle to identify and determine the current image information of one or more objects to be prompted in front (such as other motor vehicles, pedestrians, non-motor vehicles), the relative delay from collection to processing of the above information, and the real-time relative speed between the object to be prompted and the vehicle. There is no restriction here.

[0178] The first image information is the image information of at least one object to be prompted in the vehicle coordinate system, which can be understood as the image information of the object to be prompted captured by the sensor and positioned and described in the vehicle coordinate system. The use of the vehicle coordinate system ensures the direct correspondence between the image information and the driver's line of sight, which is convenient for subsequent display on the AR-HUD client, and is not limited here.

[0179] The relative delay includes the dynamic delay of the first image information during the acquisition, transmission and calculation process, and the fixed delay during the rendering and display process. It can be understood as the time difference from the acquisition of sensor data, transmission to the calculation and analysis of the central processing unit, and finally rendering and displaying the processed information on the AR-HUD client. For example, the camera collects perception data, and the collected perception data is calculated and output. There is a dynamic delay in this process. The output perception data is rendered and displayed in the ARHUD field of view. There is a fixed delay in this process, which is not limited here.

[0180] Based on the relative speed vector and the relative time delay, the time delay compensation of the first image information is performed to obtain the second image information. It can be understood that the relative speed vector information of the object to be prompted and the vehicle is used, combined with the relative time delay of the first image information, and the first image information is adjusted through an algorithm to predict and compensate for the difference between the image and the actual position caused by data processing and display delays, thereby generating more real-time and accurate second image information, which is not limited here.

[0181] The second image information is the image information of at least one object to be prompted in the vehicle coordinate system. It can be understood that the image information after delay compensation is still described in the vehicle coordinate system, but more accurately reflects the actual position and state of the object to be prompted at the time of display. The compensated image information can ensure that the content displayed by the AR-HUD client is synchronized with the real driving scene, thereby improving the real-time and effectiveness of driving assistance, which is not restricted here.

[0182] In an embodiment of the present invention, first, the first image information of the object to be prompted in front and the relative speed vector between this information and the vehicle are determined based on the sensor data, and the relative delay of information processing (acquisition, transmission, calculation) and rendering display is calculated. Then, the first image information is delayed compensated using the relative speed vector and the relative delay to eliminate the time difference from data acquisition to display, and generate second image information that is closer to real time. Finally, the compensated second image information in the vehicle coordinate system is accurately displayed within the field of view of the AR-HUD client, so that the prompt of the object to be prompted seen by the driver is synchronized with the actual environment, thereby significantly improving driving safety and driving experience in low visibility environments or states.

[0183] Optionally, determining the relative speed vector between at least one object to be prompted and the vehicle based on the environmental perception data comprises the following steps:

[0184] Determine first driving data of the vehicle and second driving data of at least one object to be prompted based on the environmental perception data, wherein the first driving data includes a current speed and a heading angle of the vehicle in the vehicle coordinate system, and the second driving data includes a current speed and a driving direction of at least one object to be prompted in the world coordinate system;

[0185] Convert the first driving data into a world coordinate system to obtain third driving data;

[0186] Determine a first relative velocity vector between the vehicle and at least one object to be prompted in a world coordinate system based on the second driving data and the third driving data;

[0187] The first relative velocity vector is converted to the vehicle coordinate system to obtain a relative velocity vector, wherein the relative velocity vector includes relative velocity information and relative direction information.

[0188] In the embodiment of the present invention, the first driving data can be understood as the driving data of the vehicle itself in the vehicle coordinate system, mainly including the current speed and heading angle of the vehicle, which is used to describe the real-time motion state and direction of the vehicle, and is not limited here.

[0189] The second driving data of at least one object to be prompted can be understood as the driving data of the object to be prompted (such as other motor vehicles, pedestrians, bicycles, etc.) in the world coordinate system, including the current speed and driving direction of the object to be prompted, which is not limited here.

[0190] The third driving data may be understood as data obtained by converting the first driving data of the vehicle from the vehicle coordinate system to the world coordinate system.

[0191] The first relative speed vector may be understood as a relative speed vector between the vehicle and each object to be prompted calculated based on the second driving data and the third driving data, which is not limited here.

[0192] Determining the first driving data of the vehicle and the second driving data of at least one object to be prompted based on environmental perception data can be understood as using sensors installed on the vehicle to collect data to determine the driving state of the vehicle (first driving data, including speed and heading angle, etc.) and the driving state of at least one object to be prompted (such as other vehicles, pedestrians) (second driving data, including speed and driving direction, etc.), which is not limited here.

[0193] Converting the first driving data to the world coordinate system to obtain the third driving data can be understood as converting the first driving data of the vehicle from the vehicle coordinate system to a global reference system, namely, the world coordinate system, so as to compare and calculate with the second driving data of the object to be prompted in the same coordinate system, which is not limited here.

[0194] Determining the first relative velocity vector between the vehicle and at least one object to be prompted in the world coordinate system based on the second driving data and the third driving data can be understood as calculating the relative speed and direction between the vehicle and the object to be prompted in the world coordinate system using the third driving data of the vehicle and the second driving data of the object to be prompted, which is not limited here.

[0195] The first relative velocity vector is converted to the vehicle coordinate system to obtain the relative velocity vector. It can be understood that in order to more intuitively show the relative motion of the object to be prompted to the driver, the first relative velocity vector in the world coordinate system is converted back to the vehicle coordinate system so that it can be presented from the driver's perspective on the AR-HUD client to provide more direct driving assistance information. This is not limited here.

[0196] In an embodiment of the present invention, first, various sensors are used to obtain the first driving data of the vehicle (including the speed and heading angle in the vehicle coordinate system) and the second driving data of at least one object to be prompted (including the speed and direction in the world coordinate system). Then, the first driving data of the vehicle is converted to the world coordinate system to form the third driving data, which realizes the coordinate unification with the data of the object to be prompted, which is convenient for subsequent calculations. Then, based on the second driving data and the third driving data in the unified coordinate system, the first relative speed vector between the vehicle and the object to be prompted is calculated, that is, the speed and direction of the object to be prompted relative to the vehicle is described. Finally, the first relative speed vector is converted back to the vehicle coordinate system to obtain the relative speed vector, which contains the speed information and direction information of the object to be prompted relative to the driver's field of view, so as to facilitate the display on the AR-HUD client from the driver's most direct perspective, thereby providing the driver with a more real-time and accurate motion status prompt of the object to be prompted in front in a complex driving environment (such as rainy and foggy weather), significantly improving driving safety and driving experience.

[0197] Optionally, performing delay compensation on the first image information based on the relative speed vector and the relative delay to obtain the second image information includes the following steps:

[0198] Determine a relative distance between the vehicle and at least one object to be prompted based on a relative velocity vector and a relative time delay;

[0199] Based on the relative distance and relative direction information, time delay compensation is performed on the first image information in the vehicle coordinate system to obtain second image information, wherein the time delay compensation is used to adjust the coordinates of the first image information to obtain the second image information.

[0200] In the embodiment of the present invention, the relative distance may be understood as the distance between at least one object to be prompted and the vehicle, which is calculated based on the relative speed vector and the relative time delay and is not limited here.

[0201] The relative direction information can be understood as the direction of the object to be prompted observed from the vehicle coordinate system, which is determined based on the relative velocity vector and is not limited here.

[0202] Determining the relative distance between the vehicle and at least one object to be prompted based on the relative speed vector and the relative time delay can be understood as calculating the actual position of the object to be prompted relative to the vehicle at the display time using the relative speed information between the vehicle and the object to be prompted and the relative time delay in system processing (from data acquisition to display), which is not limited here.

[0203] Based on the relative distance and relative direction information, the first image information is compensated for the delay in the self-vehicle coordinate system to obtain the second image information. It can be understood that the relative distance and relative direction information are used to adjust the position and direction of the first image information of the object to be prompted in the field of view of the self-vehicle to compensate for the delay in system processing and improve the synchronization and accuracy of the AR-HUD client display information. This is not limited here.

[0204] Delay compensation is used to adjust the coordinates of the first image information to obtain the second image information. It can be understood that delay compensation is used to correct the coordinate position of the first image information (i.e., the information of the object to be prompted displayed by the original sensor data) under the influence of delay, ensuring that the information of the object to be prompted (second image information) displayed on the AR-HUD client matches the position and direction in the actual environment, thereby realizing real-time updating and accurate display of the information of the object to be prompted, thereby improving the effectiveness of the driving assistance system and driving safety, which is not restricted here.

[0205] In the embodiment of the present invention, firstly, based on the relative speed vector between the vehicle and the object to be prompted, combined with the relative delay of data processing, the real-time relative distance of the object to be prompted relative to the vehicle at the time of display is predicted and calculated. Subsequently, the original captured first image information is adjusted using the obtained relative distance and the relative direction information of the object to be prompted, i.e., delay compensation, to ensure that the displayed image of the object to be prompted matches the actual field of view of the driver, and finally generate more accurate second image information.

[0206] Optionally, the augmented reality head-up display method of vehicle auxiliary element information further includes the following steps:

[0207] If the signal data is in the target state, record the duration of the signal data in the target state;

[0208] In response to the duration being greater than a preset duration, it is determined that the vehicle is currently in a low visibility environment.

[0209] In the embodiment of the present invention, the target state can be understood as a specific signal data state related to the low visibility environment detected by the intelligent automobile system through various sensors (such as wiper sensors, fog light sensors, cameras, etc.). Exemplarily, the target state may include: the wiper is in an activated state, and the running speed exceeds a certain threshold, indicating that it is raining outside, and the rainfall is heavy, which may affect the driving field of vision; the fog light is activated, suggesting that there is fog or smoke in the external environment, resulting in reduced visibility; the image data analysis captured by the camera shows that the visibility index (such as visible light intensity, image contrast, etc.) is lower than the preset threshold, indicating that the current environment visibility is low; the meteorological parameters such as humidity or air pressure reported by the meteorological sensor exceed the preset range, which may mean that rainy and foggy weather is coming, which is not limited here.

[0210] If the signal data is in the target state, the duration of the signal data in the target state can be understood as the vehicle detecting specific signals related to the low visibility environment (such as active wipers, fog lights on, camera images showing low visibility, etc.) through its onboard sensors (such as wiper sensors, fog light sensors, cameras, etc.), and the vehicle will start timing to record the length of time these signals continue to display low visibility related conditions. This helps the vehicle determine whether the current environmental conditions have really reached the low visibility standard and avoid misjudgment due to short-term raindrops, fog, etc. The accumulation of duration is one of the important bases for the vehicle to make accurate environmental assessments, and is not restricted here.

[0211] In response to the duration being greater than the preset duration, determining that the vehicle is currently in a low visibility environment can be understood as when the signal data continues to maintain the target state for a period exceeding the preset threshold (i.e., the preset duration), the vehicle believes that the current environment has indeed entered a low visibility state. The preset duration is set to ensure the accuracy of the vehicle's judgment and avoid mistakenly starting the low visibility mode due to occasional environmental changes or short-term events (such as spraying glass water). When the duration exceeds the preset duration, the vehicle will formally confirm that the vehicle is in a low visibility environment and activate the low visibility driving assistance function of the AR-HUD client accordingly, providing the driver with clearer and more intuitive driving information and enhancing driving safety, which is not restricted here.

[0212] In the embodiment of the present invention, the vehicle monitors specific signal data (such as wiper activity, fog light status, etc.), and when these signal data reach a preset target state, it starts recording the duration of maintaining the state. If the duration exceeds a preset threshold, the current environment is considered to be low visibility, and corresponding driving assistance functions are activated accordingly, such as the enhanced display mode of the AR-HUD client, to improve driving safety and visibility.

[0213] Optionally, the signal data includes a wiper activation signal, a wiper operating speed, a fog lamp activation signal, and rain light sensor data, and the method for displaying vehicle assisted driving information further includes the following steps:

[0214] In response to the wiper activation signal being in an activated state, the wiper operating speed being greater than a preset speed, the fog lamp activation signal being in an activated state, the rain / light sensor data indicating that the rainfall is greater than a preset rainfall amount and the light intensity is greater than a preset light intensity, it is determined that the signal data is in a target state.

[0215] In the embodiment of the present invention, the wiper activation signal can be understood as a signal that the wiper system is activated by the driver or the automatic system. For example, when the driver turns on the wiper or the automatic wiper function of the vehicle detects raindrops, the vehicle will send an activation signal to keep the wiper in a working state, which is not limited here.

[0216] The wiper running speed can be understood as the speed at which the wiper runs after it is started. For example, the wiper speed can be low, medium or high, thereby reflecting the density of raindrops and the intensity of rainfall. The wiper running speed data can be provided by the controller of the wiper system to assist in determining whether it is in a rainy and foggy environment and the severity of the rainy and foggy environment, which is not limited here.

[0217] The fog light activation signal can be understood as a light signal that provides the driver with better forward visibility under low visibility conditions, such as fog, rain, and snow. For example, when the fog light is activated, the vehicle will send a signal indicating that the current environment may have poor visibility and additional lighting is needed to improve the field of vision, which is not limited here.

[0218] The rain and light sensor data can be understood as the data collected by the light sensor installed on the vehicle (which may also be integrated with the rain sensor function), which is used to monitor the external light intensity and rainfall. When the data shows that the rainfall is greater than the preset threshold and the light intensity is lower than the preset threshold, it means that the current environment is rainy and the light is dim, and the visibility is low, and there is no restriction here.

[0219] In response to the wiper activation signal being activated, the wiper operating speed being greater than a preset speed, the fog lamp activation signal being activated, and the rain / light amount sensor data indicating that the rainfall is greater than a preset rainfall amount and the light intensity is greater than a preset light intensity, determining that the signal data is in the target state can be understood as the vehicle monitoring a series of specific sensor signals and data, including that the wipers are running (the wiper activation signal is activated), the wiper operating speed exceeds a set threshold (preset speed), the fog lamps are turned on (the fog lamp activation signal is activated), and the external rainfall level reported by the rain / light amount sensor is higher than a preset rainfall amount threshold, and the light intensity is also higher than a preset light intensity threshold. When all of the above conditions are met at the same time, the system considers that the current environmental characteristics of the vehicle meet the specific standards of low visibility, that is, the vehicle is in the target state, which is not restricted here.

[0220] In the embodiment of the present invention, through the above-mentioned judgment mechanism and the comprehensive evaluation of multiple sensor data, severe weather conditions such as rain, fog, heavy rainfall, etc. can be identified more accurately, avoiding the misjudgment that may be caused by a single sensor, and providing a reliable trigger basis for the subsequent activation of the AR-HUD client's low-visibility driving assistance function, thereby ensuring that the driver can obtain enhanced visual information in a low-visibility environment and improve driving safety.

[0221] Optionally, obtaining signal data recognized by the vehicle through a sensor includes the following steps:

[0222] Acquire first signal data collected by the vehicle through at least one sensor;

[0223] The first signal data is fused to obtain signal data.

[0224] In the embodiment of the present invention, the first signal data may be understood as data information directly reflecting the surrounding environment of the vehicle and the state of the vehicle itself, which is acquired from a plurality of sensors in the vehicle without being processed or analyzed, and is not limited here.

[0225] Fusion processing can be understood as integrating and analyzing the first signal data using a fusion algorithm to form more accurate, comprehensive, and reliable target sensing data. For example, algorithms and techniques for fusion processing include Kalman filtering, deep learning models, Bayesian networks, and signal weighted fusion, which are not limited here.

[0226] Acquiring the first signal data collected by the vehicle through at least one sensor can be understood as collecting or reading the raw data directly output by various sensors installed on the vehicle. Exemplarily, each sensor is responsible for monitoring a specific type of information, such as the external environment image captured by the camera, the distance and speed of the front object measured by the millimeter-wave radar, and the dynamic changes of the vehicle recorded by the inertial measurement unit. The first signal data is not processed in any way and directly reflects the information measured by the sensor at the current time point. It is the basis for subsequent fusion processing and analysis and is not limited here.

[0227] The first signal data is fused to obtain signal data, which can be understood as comprehensively analyzing and processing the first signal data of the multiple sensors collected above to obtain more accurate, complete and reliable sensor data. Exemplarily, the information of different sensors is combined through fusion algorithms and technologies to eliminate redundancy, correct errors, and fill blind spots, thereby providing a comprehensive environmental perception model, which is not limited here.

[0228] In an embodiment of the present invention, by fusing the first signal data, the augmented reality auxiliary element prompt mode can better understand complex driving environments, such as the position, speed, lane line status, etc. of the vehicle ahead in rainy and foggy weather, thereby providing more accurate augmented reality display and driving assistance information and improving driving safety.

[0229] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0230] According to an embodiment of the present invention, an embodiment of an augmented reality head-up display device for vehicle auxiliary element information is provided. It should be noted that the device can be used to execute the augmented reality head-up display method for vehicle auxiliary element information.

[0231] Figure 5 is a schematic diagram of an augmented reality head-up display device for vehicle auxiliary element information according to an embodiment of the present invention, such as Figure 5As shown, the augmented reality head-up display 500 of vehicle auxiliary element information includes: a first acquisition module 501, which is used to obtain signal data recognized by the vehicle through the sensor; a first determination module 502, which is used to determine the current operating state of the augmented reality display client if it is determined that the signal data indicates that the vehicle is currently in a low visibility environment; a control module 503, which is used to turn on the augmented reality display client if the augmented reality display client is in a closed state, and directly enter the augmented reality auxiliary element prompt mode for display; or, if the augmented reality display client is already running and in the first display mode state, the first display mode state is switched to the augmented reality auxiliary element prompt mode and displayed; a second acquisition module 504, which is used to obtain the environmental perception data of the vehicle in the augmented reality auxiliary element prompt mode; a second determination module 505, which is used to determine the auxiliary element information based on the environmental perception data; a display module 506, which is used to display the auxiliary element information at a corresponding position within the field of view of the augmented reality head-up display, wherein the auxiliary element information includes a prompt image of the object to be prompted.

[0232] Furthermore, the auxiliary element information also includes a lane line image of the current lane where the vehicle is located. The second acquisition module 504 is also used to acquire the lane line image of the current lane based on the environmental perception data, and render the lane line image of the current lane at a corresponding position within the field of view of the augmented reality head-up display.

[0233] Furthermore, the auxiliary element information also includes a lane line image of the lane to be traveled, and the lane line image of the lane to be traveled includes the lane line image of the current lane where the vehicle is located and the lane line image of the lane to which the vehicle is about to change lanes. The second acquisition module 504 is also used to obtain lane-level light carpet control line data in the map coordinate system from the cloud server; determine the lane to be traveled by the vehicle based on the lane-level light carpet control line data, wherein the lane to be traveled includes the current lane where the vehicle is located and the lane to which the vehicle is about to change lanes; determine first lane line element information corresponding to the lane to be traveled from the environmental perception data; convert the first lane line element information from the world coordinate system to the vehicle coordinate system to obtain second lane line element information in the vehicle coordinate system; fuse the second lane line element information and the lane-level light carpet control line data to obtain third lane line element information in the vehicle coordinate system; and determine the lane line image of the lane to be traveled based on the third lane line element information.

[0234] Furthermore, the auxiliary element information also includes a lane line image of the lane to be traveled, and the lane line image of the lane to be traveled includes the lane line image of the current lane where the vehicle is located and the lane line image of the lane to which the vehicle is about to change lanes. The second acquisition module 504 is also used to calculate the lane-level light blanket control line data in the vehicle coordinate system according to the navigation direction data and the environmental perception data; determine the lane to be traveled by the vehicle based on the lane-level light blanket control line data, wherein the lane to be traveled includes the current lane where the vehicle is located and the lane to which the vehicle is about to change lanes; determine the lane line element information corresponding to the lane to be traveled from the environmental perception data; and determine the lane line image of the lane to be traveled based on the lane line element information.

[0235] Furthermore, the signal data includes external sensor data of the vehicle and internal sensor data of the vehicle. The first determination module 502 is also used to determine that the vehicle is currently in a low visibility environment if the external sensor data of the vehicle indicates that the vehicle is currently in at least one of a heavy rain environment, a smoke environment, a haze environment, a blizzard environment, a sandstorm environment, and a strong light environment; if the internal sensor data of the vehicle indicates that the front windshield window of the vehicle is fogged or the front windshield is suddenly blocked, it is determined that the vehicle is currently in a low visibility environment.

[0236] Furthermore, the auxiliary element information also includes at least one of the following: a road edge line image, an image of a lane change warning light of an object to be prompted, a bird's-eye view image of the vehicle itself, and a side and rear steering image, wherein the lane change warning image of the object to be prompted is the image information of the lane change warning light of the vehicle in front, the bird's-eye view image of the vehicle itself is the image information from the bird's-eye view perspective of the vehicle itself, and the side and rear steering image is the image information from the side and rear of the vehicle itself.

[0237] Furthermore, the display module 506 is also used to obtain the first position of the object to be prompted ahead in the current frame; estimate the position of the object to be prompted ahead in at least one next frame in the world coordinate system based on the relative speed between the object to be prompted ahead and the vehicle, and obtain at least one second position; perform window averaging on at least one second position to obtain a third position; convert the third position to the vehicle coordinate system to obtain a fourth position; and display a prompt image of the object to be prompted or an image of a lane change warning light of the object to be prompted at the fourth position within the field of view of the augmented reality head-up display.

[0238] Furthermore, the augmented reality head-up display device of vehicle auxiliary element information also includes: a compensation module, which is used to determine first image information of at least one object to be prompted in front of the vehicle, a relative delay corresponding to the first image information, and a relative speed vector between at least one object to be prompted and the vehicle based on environmental perception data, wherein the first image information is image information of at least one object to be prompted in the vehicle coordinate system, and the relative delay includes a dynamic delay in the acquisition, transmission and calculation of the first image information, and a fixed delay in the rendering and display process; based on the relative speed vector and the relative delay, the first image information is delayed compensated to obtain second image information, so as to display the second image information at a corresponding position within the field of view of the augmented reality head-up display, wherein the second image information is image information of at least one object to be prompted in the vehicle coordinate system.

[0239] Furthermore, the compensation module is also used to determine first driving data of the vehicle and second driving data of at least one object to be prompted based on the environmental perception data, wherein the first driving data includes the current speed and heading angle of the vehicle in the vehicle coordinate system, and the second driving data includes the current speed and driving direction of at least one object to be prompted in the world coordinate system; convert the first driving data to the world coordinate system to obtain third driving data; determine a first relative velocity vector between the vehicle and at least one object to be prompted in the world coordinate system based on the second driving data and the third driving data; convert the first relative velocity vector to the vehicle coordinate system to obtain a relative velocity vector, wherein the relative velocity vector includes relative velocity information and relative direction information.

[0240] Furthermore, the compensation module is also used to determine the relative distance between the vehicle and at least one object to be prompted based on the relative velocity vector and the relative time delay; based on the relative distance and relative direction information, the first image information is delayed compensated in the vehicle coordinate system to obtain the second image information, wherein the time delay compensation is used to adjust the coordinates of the first image information to obtain the second image information.

[0241] Furthermore, the first determination module 502 is also used to record the duration of the signal data being in the target state if the signal data is in the target state; in response to the duration being greater than a preset duration, determine that the vehicle is currently in a low visibility environment.

[0242] Furthermore, the first determination module 502 is also used to determine that the signal data is in the target state in response to the wiper activation signal being in an activated state, the wiper operating speed being greater than a preset speed, the fog lamp activation signal being in an activated state, the rain light sensor data indicating that the rainfall is greater than a preset rainfall amount and the light intensity is greater than a preset light intensity.

[0243] Furthermore, the first acquisition module 501 is also used to acquire first signal data collected by the vehicle through at least one sensor; and perform fusion processing on the first signal data to obtain signal data.

[0244] According to another aspect of an embodiment of the present invention, a vehicle is further provided, and the vehicle is used to execute the augmented reality head-up display method for implementing the vehicle auxiliary element information in various embodiments of the present invention.

[0245] According to another aspect of an embodiment of the present invention, an electronic device is also provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the augmented reality head-up display method of vehicle auxiliary element information in various embodiments of the present invention.

[0246] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, the computer-readable storage medium including a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the augmented reality head-up display method of vehicle auxiliary element information in various embodiments of the present invention.

[0247] According to another aspect of an embodiment of the present invention, a computer program product is also provided, including a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the augmented reality head-up display method of vehicle auxiliary element information in each embodiment of the present invention is implemented.

[0248] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0249] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0250] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0251] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0252] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk and other media that can store program codes.

[0253] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. An augmented reality head-up display method for vehicle auxiliary element information, characterized in that: include: Obtain signal data recognized by the vehicle through sensors; If it is determined that the signal data indicates that the vehicle is currently in a low visibility environment, determining a current operating state of the augmented reality display client; If the augmented reality display client is in a closed state, the augmented reality display client is turned on, and the augmented reality auxiliary element prompt mode is directly entered for display; Alternatively, if the augmented reality display client is already running and in the first display mode state, the first display mode state is switched to the augmented reality auxiliary element prompt mode and displayed; In the augmented reality auxiliary element prompt mode, acquiring environmental perception data of the vehicle; Determining auxiliary element information based on the environmental perception data; The auxiliary element information is displayed at a corresponding position within the field of view of the augmented reality head-up display, wherein the auxiliary element information includes a prompt image of the object to be prompted.

2. The method according to claim 1, characterized in that The auxiliary element information also includes a lane line image of the current lane where the vehicle is located, and the auxiliary element information determined based on the environmental perception data includes: A lane line image of the current lane is acquired based on the environmental perception data, and the lane line image of the current lane is rendered at a corresponding position within the augmented reality head-up display field of view.

3. The method according to claim 1, characterized in that The objects to be prompted include static objects or dynamic objects, and the dynamic objects include at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian or an animal within the augmented reality head-up display field of view of the vehicle; the static objects include at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian or an animal within the augmented reality head-up display field of view of the vehicle.

4. The method according to claim 1, characterized in that: The auxiliary element information also includes a lane line image of the lane to be driven, and the lane line image of the lane to be driven includes a lane line image of the current lane where the vehicle is located and a lane line image of the lane to be changed to, and the auxiliary element information determined based on the environmental perception data also includes: Obtain lane-level light blanket control line data in the map coordinate system from the cloud server; Determining a lane to be driven by the ego vehicle based on the lane-level light blanket control line data, wherein the lane to be driven includes a current lane where the ego vehicle is located and a lane to be changed by the ego vehicle; Determining first lane line element information corresponding to the lane to be driven from the environmental perception data; Convert the first lane line element information from the world coordinate system to the vehicle coordinate system to obtain the second lane line element information in the vehicle coordinate system; The second lane line element information and the lane-level light blanket control line data are merged to obtain third lane line element information in the vehicle coordinate system; A lane line image of the lane to be driven is determined based on the third lane line element information.

5. The method according to claim 1, characterized in that: The auxiliary element information also includes a lane line image of the lane to be driven, and the lane line image of the lane to be driven includes a lane line image of the current lane where the vehicle is located and a lane line image of the lane to be changed to, and the auxiliary element information determined based on the environmental perception data also includes: Calculating lane-level light blanket control line data in the vehicle coordinate system according to the navigation direction data and the environment perception data; Determining a lane to be driven by the ego vehicle based on the lane-level light blanket control line data, wherein the lane to be driven includes a current lane where the ego vehicle is located and a lane to be changed by the ego vehicle; Determining lane line element information corresponding to the lane to be driven from the environmental perception data; A lane line image of the lane to be driven is determined based on the lane line element information.

6. The method according to any one of claims 1 to 5, characterized in that The signal data includes external sensor data of the vehicle and internal sensor data of the vehicle, and the method further includes: If the vehicle's external sensor data indicates that the vehicle is currently in at least one of a heavy rain environment, a smog environment, a haze environment, a snowstorm environment, a sandstorm environment, and a strong light environment, then it is determined that the vehicle is currently in the low visibility environment; If the internal sensor data of the own vehicle indicates that at least one of the front windshield window of the own vehicle is fogged or the front windshield is suddenly blocked, it is determined that the own vehicle is currently in the low visibility environment.

7. The method according to any one of claims 1 to 5, characterized in that The auxiliary element information also includes at least one of the following: a road edge line image, an image of a lane change warning light of an object to be prompted, a bird's-eye view image of the vehicle, and a side and rear steering image, wherein the lane change warning image of the object to be prompted is the image information of the lane change warning light of the vehicle in front, the bird's-eye view image of the vehicle is the image information from the bird's-eye view perspective of the vehicle, and the side and rear steering image is the image information of the side and rear of the vehicle.

8. The method according to any one of claims 1 to 5, characterized in that The displaying of the auxiliary element information at a corresponding position within the field of view of the augmented reality head-up display includes: Get the first position of the object to be prompted in the current frame; Based on the relative speed between the front object to be prompted and the vehicle, a position of the front object to be prompted in at least one next frame is estimated in a world coordinate system to obtain at least one second position; Performing window averaging on the at least one second position to obtain a third position; Convert the third position to the vehicle coordinate system to obtain a fourth position; A prompt image of the object to be prompted or an image of a lane change warning light of the object to be prompted is displayed at the fourth position within the augmented reality head-up display field of view.

9. The method according to any one of claims 1 to 5, characterized in that The method comprises: Determine, based on the environmental perception data, first image information of at least one object to be prompted in front of the vehicle, a relative delay corresponding to the first image information, and a relative speed vector between the at least one object to be prompted and the vehicle, wherein the first image information is image information of the at least one object to be prompted in the vehicle coordinate system, and the relative delay includes a dynamic delay of the first image information in the process of acquisition, transmission and calculation, and a fixed delay in the process of rendering and display; The first image information is delayed compensated based on the relative velocity vector and the relative delay to obtain second image information, so as to display the second image information at a corresponding position within the field of view of the augmented reality head-up display, wherein the second image information is image information of the at least one object to be prompted in the vehicle coordinate system.

10. The method according to claim 9, characterized in that Determining the relative speed vector between the at least one object to be prompted and the vehicle based on the environmental perception data includes: Determine first driving data of the ego vehicle and second driving data of the at least one object to be prompted based on the environmental perception data, wherein the first driving data includes a current speed and a heading angle of the ego vehicle in an ego vehicle coordinate system, and the second driving data includes a current speed and a driving direction of the at least one object to be prompted in a world coordinate system; Converting the first driving data into a world coordinate system to obtain third driving data; Determine a first relative velocity vector between the vehicle and the at least one object to be prompted in a world coordinate system based on the second driving data and the third driving data; The first relative velocity vector is converted into the vehicle coordinate system to obtain the relative velocity vector, wherein the relative velocity vector includes relative velocity information and relative direction information.

11. The method according to claim 10, characterized in that The performing delay compensation on the first image information based on the relative speed vector and the relative delay to obtain the second image information comprises: Determine the relative distance between the vehicle and the at least one object to be prompted based on the relative speed vector and the relative time delay; Based on the relative distance and the relative direction information, time delay compensation is performed on the first image information in the vehicle coordinate system to obtain the second image information, wherein the time delay compensation is used to adjust the coordinates of the first image information to obtain the second image information.

12. The method according to any one of claims 1 to 5, characterized in that The method further comprises: If the signal data is in the target state, record the duration of the signal data being in the target state; In response to the duration being greater than a preset duration, it is determined that the vehicle is currently in the low visibility environment.

13. The method according to claim 12, characterized in that The signal data includes a wiper activation signal, a wiper operating speed, a fog lamp activation signal, and rain light sensor data. The method further includes: In response to the wiper activation signal being in an activated state, the wiper operating speed being greater than a preset speed, the fog lamp activation signal being in an activated state, the rain light sensor data indicating that the rainfall is greater than a preset rainfall amount and the light intensity is greater than a preset light intensity, it is determined that the signal data is in the target state.

14. The method according to any one of claims 1 to 5, characterized in that The signal data obtained from the vehicle through the sensor includes: Acquire first signal data collected by the vehicle through at least one sensor; The first signal data is fused to obtain the signal data.

15. An augmented reality head-up display device for vehicle auxiliary element information, characterized in that: include: A first acquisition module is used to acquire signal data recognized by the vehicle through a sensor; A first determination module, configured to determine a current operating state of the augmented reality display client if it is determined that the signal data indicates that the vehicle is currently in a low visibility environment; A control module, configured to, if the augmented reality display client is in an off state, turn on the augmented reality display client and directly enter an augmented reality auxiliary element prompt mode for display; Alternatively, if the augmented reality display client is already running and in the first display mode state, the first display mode state is switched to the augmented reality auxiliary element prompt mode and displayed; A second acquisition module is used to acquire the environmental perception data of the vehicle in the augmented reality auxiliary element prompt mode; A second determination module, configured to determine auxiliary element information based on the environmental perception data; The display module is used to display the auxiliary element information at a corresponding position within the field of view of the augmented reality head-up display, wherein the auxiliary element information includes a prompt image of the object to be prompted.

16. A vehicle, characterized in that: The vehicle is used to execute the augmented reality head-up display method of vehicle auxiliary element information as described in any one of claims 1 to 14 above.

17. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the augmented reality head-up display method for vehicle auxiliary element information as described in any one of claims 1 to 14.

Images