Augmented reality head-up display method and device for vehicle auxiliary element information and electronic device

Abstract

The invention discloses an augmented reality head-up display method and device for vehicle auxiliary element information and an electronic device. The method comprises the steps that in response to the fact that a vehicle is in an augmented reality auxiliary element prompt mode, environment sensing data of the vehicle and a slope value corresponding to the vehicle are obtained, and the variable quantity of the slope value is used for reflecting the shaking state of the vehicle; auxiliary element information is determined based on the environment perception data, and the auxiliary element information comprises a prompt image of the to-be-prompted object; performing time delay compensation on the first display position of the auxiliary element information based on the gradient value to obtain a second display position; and displaying the auxiliary element information at a second display position in the augmented reality head-up display view field range of the own vehicle. The technical problem of visual malposition caused by delay when an augmented reality head-up display is adopted to display information in the prior art is solved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an augmented reality head-up display method, device, and electronic device for vehicle auxiliary element information. Background Art

[0002] To help users understand the road conditions around their vehicle while driving, Simulation Reality (SR) technology is generally used in conjunction with information displayed on the vehicle's screen to provide driving guidance. The bird's-eye view provided by the SR mode differs significantly from the driver's actual first-person perspective. In actual driving, the driver operates from a first-person perspective inside the vehicle, while the global view of the SR mode often makes the user feel disconnected from reality, reducing the intuitiveness and immersiveness of the driving assistance information. In addition, this perspective change may cause some users to feel uncomfortable, affecting the user experience. While the road condition information around the vehicle displayed on the vehicle screen is relatively accurate, it is limited by a fixed viewing angle and limited display space, and cannot provide an immersive experience that matches the driver's real-time line of sight.

[0003] To address the above-mentioned problems, no effective solutions have been proposed so far. Summary of the Invention

[0004] Embodiments of the present invention provide an augmented reality head-up display method, apparatus, and electronic device for displaying vehicle auxiliary element information, to at least solve the technical problem of visual dislocation caused by delay in displaying information using an augmented reality head-up display in related technologies.

[0005] According to one aspect of an embodiment of the present invention, a method for augmented reality head-up display of vehicle auxiliary element information is provided, comprising: in response to a vehicle being in an augmented reality auxiliary element prompt mode, obtaining environmental perception data of the vehicle and a slope value corresponding to the vehicle, wherein a change in the slope value is used to reflect a shaking state of the vehicle; determining auxiliary element information based on the environmental perception data, wherein the auxiliary element information includes a prompt image of an object to be prompted; performing time delay compensation on a first display position of the auxiliary element information based on the slope value to obtain a second display position; and displaying the auxiliary element information at the second display position within the augmented reality head-up display field of view of the vehicle.

[0006] Furthermore, based on the slope value, the first display position of the auxiliary element information is compensated for the delay to obtain the second display position, which includes: obtaining the first display position of the object to be prompted in the vehicle coordinate system, and the first slope value corresponding to the vehicle in the vehicle coordinate system; converting the first display position to the world coordinate system to obtain a third display position, and converting the first slope value to the world coordinate system to obtain a second slope value; determining the horizontal plane motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative delay and second slope value between the object to be prompted and the vehicle, wherein the horizontal plane motion vector is used to represent the motion direction of the object to be prompted and the distance offset of the object to be prompted, and the relative delay includes the dynamic delay of the prompt image of the object to be prompted during the acquisition, transmission and calculation process, as well as the fixed delay during the rendering and display process; determining the fourth display position of the object to be prompted in the world coordinate system based on the third display position and the horizontal plane motion vector; converting the fourth display position to the real-time coordinate system of the vehicle based on the six-degree-of-freedom information of the vehicle to obtain the second display position.

[0007] Furthermore, converting the first display position to the world coordinate system to obtain the third display position includes: obtaining the vehicle position information and heading angle of the vehicle in the world coordinate system; and converting the first display position to the world coordinate system based on the vehicle position information and heading angle to obtain the third display position.

[0008] Furthermore, converting the first slope value into a world coordinate system to obtain a second slope value includes: obtaining a relative angle of the vehicle relative to the ground in the world coordinate system; and converting the first slope value into the world coordinate system based on the relative angle to obtain a second slope value.

[0009] Furthermore, converting the first slope value to the world coordinate system based on the relative angle to obtain the second slope value includes: converting the first slope value to the world coordinate system based on the relative angle to obtain a third slope value; and filtering the third slope value to obtain the second slope value.

[0010] Furthermore, determining the horizontal plane motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative time delay and second slope value between the object to be prompted and the vehicle includes: obtaining a first relative speed of the object to be prompted and the vehicle in the vehicle coordinate system; converting the first relative speed to the world coordinate system to obtain a second relative speed; multiplying the second relative speed by the relative time delay to obtain a predicted motion vector of the object to be prompted in the world coordinate system; and projecting the predicted motion vector onto the horizontal plane based on the second slope value to obtain a horizontal plane motion vector.

[0011] Furthermore, converting the fourth display position to the real-time coordinate system of the vehicle based on the six-degree-of-freedom information of the vehicle to obtain the second display position includes: determining the real-time coordinate system of the vehicle based on the six-degree-of-freedom information of the vehicle; converting the fourth display position to the real-time coordinate system of the vehicle to obtain the second display position.

[0012] Furthermore, the objects to be prompted include static objects or dynamic objects. The dynamic objects include at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian or an animal within the field of view of the augmented reality head-up display of the vehicle. The static objects include at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian or an animal within the field of view of the augmented reality head-up display of the vehicle.

[0013] According to another aspect of an embodiment of the present invention, an augmented reality head-up display device for vehicle auxiliary element information is also provided, including: an acquisition module for acquiring environmental perception data of the vehicle and a slope value corresponding to the vehicle in response to the vehicle being in an augmented reality auxiliary element prompt mode, wherein the change in the slope value is used to reflect the shaking state of the vehicle; a determination module for determining auxiliary element information based on the environmental perception data, wherein the auxiliary element information includes a prompt image of an object to be prompted; a delay compensation module for performing delay compensation on a first display position of the auxiliary element information based on the slope value to obtain a second display position; and a display module for displaying the auxiliary element information at the second display position within the augmented reality head-up display field of view of the vehicle.

[0014] Furthermore, the delay compensation module is also used to obtain a first display position of the object to be prompted in the vehicle coordinate system and a first slope value corresponding to the vehicle in the vehicle coordinate system; convert the first display position to the world coordinate system to obtain a third display position, and convert the first slope value to the world coordinate system to obtain a second slope value; determine the horizontal plane motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative delay and second slope value between the object to be prompted and the vehicle, wherein the horizontal plane motion vector is used to represent the motion direction of the object to be prompted and the distance offset of the object to be prompted, and the relative delay includes the dynamic delay of the prompt image of the object to be prompted during the acquisition, transmission and calculation process, as well as the fixed delay during the rendering and display process; determine the fourth display position of the object to be prompted in the world coordinate system based on the third display position and the horizontal plane motion vector; convert the fourth display position to the real-time coordinate system of the vehicle based on the six-degree-of-freedom information of the vehicle to obtain the second display position.

[0015] Furthermore, the time delay compensation module is also used to obtain the vehicle position information and heading angle of the vehicle in the world coordinate system; based on the vehicle position information and heading angle, the first display position is converted to the world coordinate system to obtain a third display position.

[0016] Furthermore, the time delay compensation module is further configured to obtain a relative angle between the vehicle and the ground in a world coordinate system; and based on the relative angle, the first slope value is converted to the world coordinate system to obtain a second slope value.

[0017] Furthermore, the time delay compensation module is further configured to convert the first slope value into a world coordinate system based on the relative angle to obtain a third slope value; and filter the third slope value to obtain a second slope value.

[0018] Furthermore, the time delay compensation module is also used to obtain a first relative speed between the object to be prompted and the vehicle in the vehicle coordinate system; convert the first relative speed to the world coordinate system to obtain a second relative speed; multiply the second relative speed by the relative delay to obtain a predicted motion vector of the object to be prompted in the world coordinate system; and project the predicted motion vector onto the horizontal plane based on the second slope value to obtain a horizontal plane motion vector.

[0019] Furthermore, the time delay compensation module is also used to determine the real-time coordinate system of the vehicle based on the six-degree-of-freedom information of the vehicle; and convert the fourth display position into the real-time coordinate system of the vehicle to obtain the second display position.

[0020] Furthermore, the display module is also used to display that the objects to be prompted include static objects or dynamic objects. The dynamic objects include at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian or an animal within the augmented reality head-up display field of view of the vehicle. The static objects include at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian or an animal within the augmented reality head-up display field of view of the vehicle.

[0021] According to another aspect of an embodiment of the present invention, a vehicle is further provided, and the vehicle is used to execute the augmented reality head-up display method of vehicle auxiliary element information in various embodiments of the present invention.

[0022] According to another aspect of an embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the augmented reality head-up display method of vehicle auxiliary element information in various embodiments of the present invention.

[0023] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the augmented reality head-up display method of vehicle auxiliary element information in various embodiments of the present invention.

[0024] According to another aspect of an embodiment of the present invention, a computer program product is further provided, including a computer program. When the computer program is executed by a processor, the computer program implements the augmented reality head-up display method of vehicle auxiliary element information in various embodiments of the present invention.

[0025] In an embodiment of the present invention, a combination of dynamic slope compensation and time delay correction is employed to accurately calculate the change in slope value to reflect the vehicle's shaking state. Auxiliary element information, including a prompt image of the object to be prompted, is determined based on environmental perception data. The auxiliary element information's first display position is then subjected to real-time time delay compensation based on the slope value, resulting in a more accurate second display position. Displaying the auxiliary element information at the second display position within the vehicle's ARHUD field of view ensures the immediacy and accuracy of the information, reducing ARHUD display delays and improving the match between displayed content and the actual environment. This significantly improves driving safety and user experience, further resolving the technical issue of visual misalignment caused by delays in displaying information using augmented reality head-up displays in related technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0027] Figure 1 This is a schematic diagram of a self-vehicle and a neighboring vehicle based on simulated reality according to the prior art;

[0028] Figure 2 This is a schematic diagram of displaying road driving information on a vehicle screen according to the prior art;

[0029] Figure 3 is a schematic diagram of displaying vehicle surrounding information based on an augmented reality head-up display according to an embodiment of the present invention;

[0030] Figure 4 1 is a flow chart of an augmented reality head-up display method for vehicle auxiliary element information according to an embodiment of the present invention;

[0031] Figure 5 is a schematic diagram of adjusting display content based on slope information according to an embodiment of the present invention;

[0032] Figure 6 2 is a schematic diagram of an augmented reality head-up display device for vehicle auxiliary element information according to an embodiment of the present invention. DETAILED DESCRIPTION

[0033] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first", "second", etc. in the description and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0035] Figure 1 This is a schematic diagram of the vehicle and the adjacent vehicle based on the simulation reality according to the existing technology, such as Figure 1 As shown, the ego vehicle and the adjacent vehicle are displayed in the same frame, and the relative relationship between the ego vehicle and the adjacent vehicle is correct. However, the bird's-eye view provided by the SR mode is significantly different from the driver's actual first-person perspective, so the user's physical perception is not strong. In addition, in the SR mode, the relative positions and motion states between vehicles are generally correct. However, due to the lack of precise tracking of the real-time dynamics of individual vehicles, especially in high-speed driving or emergency situations, this bird's-eye view may not provide sufficiently detailed information. In addition, this perspective conversion may make some users feel uncomfortable, affecting the user experience. Figure 2 This is a schematic diagram of displaying road driving information on a vehicle screen according to the prior art. Figure 2 As shown in the figure, although the road condition information around the vehicle displayed on the car screen is relatively accurate, the car screen may display multiple information at the same time, such as navigation instructions, entertainment system control, vehicle status, etc. Excessive or overly dense information presentation sometimes makes it difficult for the driver to quickly filter out key information, which increases the cognitive burden.

[0036] In order to solve the problems existing in the above-mentioned simulated reality technology and car screen displays, augmented reality head-up display (ARHUD) has been proposed as an emerging head-up display technology, which is mainly used to guide users to identify and judge path information such as the road ahead, obstacles, traffic signals, etc. Figure 3 FIG. 1 is a schematic diagram of displaying vehicle surrounding information based on an augmented reality head-up display according to an embodiment of the present invention. Figure 3 As shown, ARHUD is an innovative driving assistance technology that can overlay virtual information (such as navigation instructions, vehicle status, road condition warnings, weather information, etc.) on the real-world scene directly in front of the driver through the windshield or a dedicated transparent display. The key advantage of ARHUD is its ability to seamlessly integrate virtual information with the driver's real line of sight, providing intuitive and safe driving assistance and reducing the driver's attention diversion. However, when using ARHUD to provide driving assistance to users, the main difficulty is how to compensate for delays, that is, how to ensure that the displayed information is synchronized with the dynamics of the real world to avoid visual dislocation caused by delays.

[0037] Although both using the car screen to display navigation information during vehicle driving and displaying the navigation information through AR HUD can provide navigation guidance, displaying navigation information through AR HUD does not use the same navigation information in place of a different display subject. Instead, there are essential differences in the generation and implementation logic and effects of the displayed navigation information. The difference between the two determines their respective characteristics and application scenarios.

[0038] The core of AR HUD lies in the integration of the navigation guidance light carpet with the actual road. This integration combines the perceived lane markings ahead, surrounding vehicles, non-motorized vehicles, pedestrians, and animals, and other necessary information with the real environment, presenting it on the windshield in front of the driver's line of sight. Effectively, this alignment of virtual navigation information with the real world maintains a high degree of consistency between virtual information and real-world elements, providing users with an intuitive and immersive driving assistance experience.

[0039] In contrast, the navigation information displayed on the car screen includes the navigation route (not the navigation guidance light carpet), the vehicle itself, and surrounding vehicles, and other virtual information rendered through animation on a fixed display screen inside the car. The navigation route and virtual information do not need to be spatially integrated with the external environment. Their main purpose is to provide users with a general understanding of the surrounding environment, and there is no high requirement for accuracy.

[0040] The specific differences are as follows:

[0041] Table 1

[0042]

[0043] As shown in Table 1, AR HUDs differ fundamentally from traditional in-car screen navigation displays in terms of whether virtual information aligns with the actual road. Due to its augmented reality capabilities, AR HUDs can overlay navigation information directly within the driver's field of view. This displays a navigation light carpet on the road the vehicle is currently on, aligning the navigation light carpet with the actual road. This allows users to clearly understand how to navigate the road ahead, making it highly intuitive. In contrast, in-car screen navigation does not require direct visual integration with the road and does not display navigation information via a light carpet. Instead, it color-codes the available roads ahead, indicating to the user which roads are accessible. This allows the user to know which roads are available, leaving the decision to choose which road to take. In addition, AR HUD can display information such as the lane lines ahead within a certain range ahead, motor vehicles, non-motor vehicles, pedestrians or animals in the surrounding environment, and intuitively prompt users of external factors that may affect driving operations. Especially in environments with low visibility, users may not notice what the surrounding environment is like. This information can greatly help users make decisions to avoid traffic accidents. However, the navigation on the car screen will render some surrounding objects, but it can only roughly understand that there may be an object nearby, and users cannot intuitively feel the specific location of the object.

[0044] Regarding whether it is necessary to predict the position of the vehicle in front in real time, since the underlying implementation logic of the two technologies is different, for example, assuming that in low visibility, the position element of the vehicle in front is displayed through ARHUD to help users identify the distance between the vehicle and the vehicle in front, how to ensure that this position element is consistent with the vehicle in front is very important for users. It is necessary to predict the position of the vehicle in front to be more consistent. For example, the vehicle in front suddenly accelerates or decelerates. In order to make the animation rendering on the ARHUD keep up with the vehicle in front, it is necessary to predict the position of the vehicle in front. Specifically, it is necessary to first obtain multiple predicted positions of the vehicle in front (for example, some predicted positions of the vehicle in front are the positions after acceleration, and some predicted positions of the vehicle in front are the positions after deceleration). Then, the current frame position of the vehicle in front is windowed and averaged to obtain a predicted position. At this time, this predicted position can be displayed on the ARHUD. At this time, even if the vehicle in front suddenly accelerates, the predicted position is relatively consistent.

[0045] The navigation screen displayed on the car screen will have an animated rendering of the vehicle in front, but this animation rendering does not have such a strong requirement for being able to fit the vehicle in front. It does not matter even if there is a certain distance difference between the rendered position of the vehicle in front and the actual position. The car screen only needs to indicate an approximate relative position of the vehicle in front, and does not require a particularly accurate position of the vehicle in front. In other words, the navigation displayed on the car screen does not predict the speed of the vehicle in front, but instead uses perception data to detect and calculate in real time whether the vehicle will collide with the vehicle in front at the moment, without knowing whether the vehicle will collide with the vehicle in front in the future.

[0046] Regarding whether there is a problem of navigation guidance light carpet exceeding the frame, since the display field of view of ARHUD is only the content within a certain range in front of the vehicle, and the navigation guidance light carpet must be aligned with the road conditions, for example, if you want to turn right or make a U-turn ahead, in this case, the navigation guidance light carpet will exceed the frame. However, the picture displayed on the car screen includes the passable road ahead (and the navigation guidance light carpet is not displayed). Even if you want to turn right or make a U-turn ahead, the passable road ahead will be displayed in the picture, so the car screen does not have a problem of navigation guidance light carpet exceeding the frame.

[0047] According to an embodiment of the present invention, an embodiment of an augmented reality head-up display method for vehicle auxiliary element information is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0048] Figure 4 FIG. 1 is a flow chart of an augmented reality head-up display method for vehicle auxiliary element information according to an embodiment of the present invention. Figure 4 As shown, the method includes the following steps:

[0049] Step S40 , in response to the vehicle being in the augmented reality auxiliary element prompt mode, obtaining the vehicle's environmental perception data and the slope value corresponding to the vehicle, wherein a change in the slope value is used to reflect the shaking state of the vehicle.

[0050] In embodiments of the present invention, the augmented reality assistive element prompt mode can be understood as referring to an operating mode of a vehicle's infotainment system or driver assistance system that utilizes an ARHUD to directly display key driving assistance information in the form of virtual images within the user's actual field of view, typically through a windshield or other transparent display. The augmented reality assistive element prompt mode can provide assistive information closely aligned with actual driving conditions, helping users better understand and respond to the road environment, and is not limited herein.

[0051] Environmental perception data can be understood as information collected by various sensors of the vehicle (such as cameras, radars, lidars, etc.), which is used to detect and analyze the environment around the vehicle, including but not limited to: the position, speed and acceleration of other vehicles, road conditions such as lane lines and traffic signs, obstacles such as pedestrians and bicycles, as well as weather conditions and visibility, which are not restricted here.

[0052] The slope value corresponding to the ego vehicle can be understood as the slope of the road the vehicle is on, typically expressed in degrees or percentages. The slope value corresponding to the ego vehicle reflects the degree of inclination of the road the vehicle is currently on. Slope values are particularly important in ARHUD applications, as changes in slope can affect vehicle stability and, in turn, the correct display position of auxiliary elements. This value is not restricted here.

[0053] The vehicle's jitter can be understood as a change in vehicle stability caused by uneven roads, vehicle vibration, or other factors. For example, during driving, the vehicle may experience jitter or vibration due to potholes, ramps, or wind resistance, causing changes in slope values. This can directly affect the position of auxiliary elements displayed on the ARHUD, potentially causing the displayed content to mismatch the actual environment, affecting driving safety and experience. This is not limited here.

[0054] In response to the vehicle being in the augmented reality auxiliary element prompt mode, obtaining the vehicle's environmental perception data and the slope value corresponding to the vehicle can be understood as, when the vehicle turns on the ARHUD's assisted driving mode, the vehicle system begins to collect and process the environmental perception data and the slope value corresponding to the vehicle.

[0055] The change in the slope value is used to reflect the shaking state of the vehicle. It can be understood that when the vehicle is driving, encountering roads with different slopes will cause the vehicle to react differently, thereby causing different changes in the vehicle body posture. This is not limited here.

[0056] In this embodiment of the present invention, when the vehicle is operating in augmented reality auxiliary element prompt mode, the system actively collects environmental perception data and the current slope value, specifically focusing on the change in slope value as an indicator for assessing the vehicle's vibration status. Through real-time analysis and processing of environmental and slope information, the system can more accurately adjust the display position of auxiliary elements on the ARHUD, ensuring that the virtual information accurately corresponds to the real situation even when the vehicle is vibrating or traveling on an undulating road, greatly improving the practicality and safety of the ARHUD.

[0057] Step S42: determining auxiliary element information based on the environmental perception data, wherein the auxiliary element information includes a prompt image of the object to be prompted.

[0058] In the embodiments of the present invention, auxiliary element information can be understood as key visual auxiliary information provided to the user. For example, auxiliary element information may include navigation instructions (such as turn arrows), hazard warnings (such as collision warnings), vehicle status prompts (such as vehicle speed display), road condition information (such as slippery roads), and dynamic obstacle tracking (such as real-time tracking and annotation of surrounding vehicles or pedestrians), without limitation here.

[0059] The objects to be prompted can be understood to include static objects or dynamic objects. Dynamic objects include at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian or an animal within the field of view of the augmented reality head-up display of the vehicle. Static objects include at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian or an animal within the field of view of the augmented reality head-up display of the vehicle. There is no restriction here.

[0060] The prompt image of the object to be prompted can be understood as a visual representation of auxiliary element information. It is a virtual image seen by the user in the ARHUD view, which is used to highlight or prompt important elements in the road environment and is not restricted here.

[0061] Auxiliary element information is determined based on environmental perception data, wherein the auxiliary element information includes the prompt image of the object to be prompted. It can be understood that the ARHUD system can intelligently analyze the surrounding environment and generate corresponding auxiliary display information accordingly. The auxiliary information will be superimposed on the driver's actual field of view in the form of visual images, seamlessly integrated with the real environment, and provide direct and effective driving assistance. It is not restricted here.

[0062] In an embodiment of the present invention, the ARHUD system uses data collected by vehicle sensors to generate targeted virtual assistance information and presents it to the driver in an intuitive form. This not only improves the driving experience, but also effectively reduces the driver's cognitive burden, allowing him to focus more on driving itself while improving his perception of the surrounding environment.

[0063] Step S44: performing time delay compensation on the first display position of the auxiliary element information based on the slope value to obtain a second display position.

[0064] In the embodiments of the present invention, the first display position can be understood as the initial display position calculated by the ARHUD based on the received environmental perception data and slope value information. For example, due to delays in data processing and display updates, the first display position may deviate from the actual position to a certain extent, which is not limited here.

[0065] Delay compensation can be understood as a key step in ARHUD technology, which is used to solve the delay problem of matching the displayed content with the position of objects in the real world. For example, the specific process of delay compensation may include: predicting vehicle movement: based on the vehicle's current speed, acceleration and slope values, predicting the position and posture changes of the vehicle during the display delay time. Adjusting the display position: considering the predicted displacement and posture changes, adjust the display position of the auxiliary element information to ensure that when it is finally displayed, the auxiliary element information can accurately correspond to the position of the object or event to be prompted in the real world. There is no restriction here.

[0066] The second display position can be understood as the display position of the auxiliary element information on the display device finally determined by ARHUD after delay compensation, which is not limited here.

[0067] Performing time delay compensation on the first display position of the auxiliary element information based on the slope value to obtain the second display position can be understood as, in the ARHUD system, dynamically analyzing the changes in the slope of the vehicle's travel, adjusting the initial display position of the auxiliary element information on the display screen, thereby obtaining the final position of the auxiliary element information on the display screen, which is not limited here.

[0068] In an embodiment of the present invention, the initial display position of the auxiliary element information is adjusted based on the slope value to ensure that the display position of the auxiliary element information is consistent with reality, thereby improving the accuracy and reliability of driving assistance and having a significant effect on improving driving safety.

[0069] Step S46: display the auxiliary element information at a second display position within the augmented reality head-up display field of view of the vehicle.

[0070] In the embodiments of the present invention, the augmented reality head-up display field of view can be understood as the portion of the real environment that the driver can clearly see through the ARHUD, superimposed with virtual information. For example, the augmented reality head-up display field of view can be set in the lower area of the vehicle's front windshield, or displayed through a specially designed small transparent panel (usually located above the instrument panel) to provide more intuitive and practical driving assistance, which is not limited here.

[0071] Displaying auxiliary element information at the second display position within the vehicle's augmented reality head-up display field of view can be understood as ARHUD displaying the processed auxiliary information that takes into account the vehicle's motion state (especially slope changes) at the optimized position and time within the augmented reality head-up display field of view, thereby ensuring that the auxiliary element information seen by the driver within the field of view is real-time, accurate and of direct reference value, thereby improving the driving experience and reducing the risk of accidents. This is not restricted here.

[0072] In the embodiment of the present invention, ARHUD displays auxiliary element information within the augmented reality head-up display field of view of the vehicle. It not only takes into account the processing delay of the on-board sensor data, but also pays attention to the movement characteristics of the vehicle under different road conditions, such as dynamic changes when going uphill or downhill. Through precise prediction and matching, it ensures that the display of virtual information is highly synchronized with the real environment, providing the driver with a safer, more intuitive and effective driving assistance method.

[0073] In an embodiment of the present invention, a combination of dynamic slope compensation and time delay correction is employed to accurately calculate the change in slope value to reflect the vehicle's shaking state. Auxiliary element information, including a prompt image of the object to be prompted, is determined based on environmental perception data. The auxiliary element information's first display position is then subjected to real-time time delay compensation based on the slope value, resulting in a more accurate second display position. Displaying the auxiliary element information at the second display position within the vehicle's ARHUD field of view ensures the immediacy and accuracy of the information, reducing ARHUD display delays and improving the match between displayed content and the actual environment. This significantly improves driving safety and user experience, further resolving the technical issue of visual misalignment caused by delays in displaying information using augmented reality head-up displays in related technologies.

[0074] Optionally, in step S44, performing time delay compensation on the first display position of the auxiliary element information based on the slope value to obtain a second display position includes the following steps:

[0075] Step S441, obtaining a first display position of the object to be prompted in the vehicle coordinate system and a first slope value corresponding to the vehicle in the vehicle coordinate system;

[0076] Step S442: converting the first display position into the world coordinate system to obtain a third display position, and converting the first slope value into the world coordinate system to obtain a second slope value;

[0077] Step S443: Determine the horizontal plane motion vector of the object to be prompted in the world coordinate system based on the relative speed between the object to be prompted and the ego vehicle, the relative delay, and the second slope value. The horizontal plane motion vector represents the motion direction and distance offset of the object to be prompted. The relative delay includes the dynamic delay of the prompt image of the object to be prompted during the acquisition, transmission, and calculation processes, as well as the fixed delay during the rendering and display process.

[0078] Step S444: determining a fourth display position of the object to be prompted in the world coordinate system based on the third display position and the horizontal plane motion vector;

[0079] Step S445 : Based on the six-degree-of-freedom information of the vehicle, the fourth display position is converted into the real-time coordinate system of the vehicle to obtain the second display position.

[0080] In the embodiments of the present invention, the ego-vehicle coordinate system can be understood as a local coordinate system established with the ego-vehicle as the center. In the ego-vehicle coordinate system, the position of the vehicle is regarded as the origin. It is generally used to represent the position information of objects in the vehicle's surrounding environment relative to the ego-vehicle, but this is not limited here.

[0081] The first slope value can be understood as measured or estimated in the ego-vehicle coordinate system, reflecting the slope of the road the ego-vehicle is currently on. The slope value is crucial to ARHUD display accuracy, especially when the vehicle is traveling uphill or downhill. Slope changes can affect the vehicle's posture and dynamic behavior, so this is not restricted here.

[0082] The world coordinate system can be understood as a global coordinate system used to describe absolute position information. Compared to the ego-vehicle coordinate system, the world coordinate system is unaffected by the vehicle's own motion and provides a stable reference frame, facilitating more accurate position calculations and transformations. This is not a limitation here.

[0083] The second slope value can be understood as the result of converting the first slope value to the world coordinate system. Converting the slope value from the vehicle coordinate system to the world coordinate system helps to more accurately calculate the displacement and posture changes of the object to be prompted in different coordinate systems, and is not limited here.

[0084] Relative speed can be understood as the speed of the object to be prompted relative to the vehicle. In the ARHUD scenario, relative speed is used to predict the expected position of the object to be prompted after a period of time, and is not restricted here.

[0085] The horizontal motion vector can be understood as a combination of relative velocity and relative delay (including dynamic and fixed delays). It represents the horizontal motion trajectory of the object being prompted, ignoring vertical changes (such as uphill or downhill). This vector contains the object's motion direction and the expected horizontal displacement within the time interval, facilitating position prediction on a two-dimensional plane, and is not limited here.

[0086] Six degrees of freedom (6DOF) can be understood as information describing the vehicle's motion in three-dimensional space, including three translational degrees of freedom (movement along the x, y, and z axes) and three rotational degrees of freedom (rotation around the x, y, and z axes). In ARHUD technology, 6DOF information is used to update the vehicle's position and posture in real time, ensuring precise alignment between virtual information and the real world. This is not a limitation here.

[0087] Obtaining the first display position of the object to be prompted in the ego vehicle coordinate system and the corresponding first slope value of the ego vehicle in the ego vehicle coordinate system can be understood as the ARHUD using onboard sensors and autonomous driving technology to collect and process data from the surrounding environment in real time to determine the relative position of the object to be prompted (such as the vehicle ahead, pedestrians, obstacles, etc.) in the ego vehicle coordinate system, i.e., the first display position. Simultaneously, the system also detects and records the current slope value of the ego vehicle, i.e., the first slope value.

[0088] Converting the first display position to the world coordinate system to obtain the third display position, and converting the first slope value to the world coordinate system to obtain the second slope value can be understood as, in order to make more accurate predictions and position adjustments, ARHUD converts the first display position and the first slope value in the vehicle coordinate system to a fixed world coordinate system.

[0089] Determining the horizontal plane motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative delay and second slope value between the object to be prompted and the vehicle can be understood as ARHUD taking into account multiple dynamic factors, including the relative speed of the object to be prompted and the vehicle, the relative delay required to process the display information (the sum of the dynamic delay and the fixed delay), and the second slope value, to calculate the horizontal plane motion vector of the object to be prompted in the world coordinate system.

[0090] Determining the fourth display position of the object to be prompted in the world coordinate system based on the third display position and the horizontal plane motion vector can be understood as follows: after obtaining the horizontal plane motion vector, the system predicts the exact position where the object to be prompted should be displayed in the world coordinate system after overcoming the display delay, that is, the fourth display position, based on the third display position and the horizontal plane motion vector.

[0091] Based on the six-degree-of-freedom information of the own vehicle, the fourth display position is converted into the real-time coordinate system of the own vehicle to obtain the second display position. It can be understood that in order to convert the predicted fourth display position into a form that the driver can intuitively see, the system uses the six-degree-of-freedom information of the own vehicle (including position, posture and dynamic changes) to convert the fourth display position from the world coordinate system back to the real-time coordinate system of the own vehicle.

[0092] In an embodiment of the present invention, the ARHUD obtains the initial position information of the objects to be prompted around the vehicle in the vehicle coordinate system and the current first slope value of the vehicle, converts the above information into the world coordinate system to obtain more accurate absolute position and slope information, and uses the relative speed and relative delay of the object to be prompted, combined with the second slope value, to calculate the horizontal plane motion vector of the object in the world coordinate system. Based on the third display position (i.e., the initial position of the object to be prompted in the world coordinate system) and the horizontal plane motion vector, the fourth display position of the object to be prompted after experiencing a relative delay is determined, and the fourth display position is converted back to the real-time coordinate system of the vehicle to obtain the final second display position, ensuring that the position of the virtual information displayed on the ARHUD can be highly consistent with the actual position of the object in the driver's front field of view, providing accurate and timely driving assistance.

[0093] Optionally, in step S442, converting the first display position into a world coordinate system to obtain a third display position includes the following steps:

[0094] Step S4421, obtaining the vehicle's position information and heading angle in the world coordinate system;

[0095] Step S4422: Convert the first display position to the world coordinate system based on the vehicle position information and the heading angle to obtain a third display position.

[0096] In the embodiment of the present invention, the vehicle position information can be understood as the exact position coordinates of the vehicle in the world coordinate system.

[0097] The heading angle can be understood as the angular difference between the vehicle's current direction and a fixed reference direction (such as north) in the world coordinate system. It is used to describe the vehicle's orientation or posture.

[0098] Obtaining the vehicle's position information and heading angle in the world coordinate system can be understood as ARHUD obtaining the vehicle's precise geographic location information, including its three-dimensional coordinates (longitude, latitude, altitude) in the world coordinate system and the current heading angle, through Beidou or other satellite positioning technologies and the vehicle's own sensors (such as an inertial measurement unit).

[0099] Based on the vehicle position information and heading angle, the first display position is converted to the world coordinate system to obtain the third display position. It can be understood that the ARHUD uses the acquired vehicle position information and heading angle to perform a coordinate conversion operation to convert the first display position previously determined in the vehicle coordinate system (that is, the initial position of the auxiliary element information relative to the vehicle) into the world coordinate system to obtain the third display position.

[0100] In an embodiment of the present invention, ARHUD uses the acquired vehicle position information and heading angle as the reference point and direction for conversion, and converts the first display position previously determined in the vehicle coordinate system (i.e., the position coordinates of the object to be prompted relative to the vehicle) into the world coordinate system, which is the basis for subsequent motion prediction and delay compensation.

[0101] Optionally, in step S442, converting the first slope value into a world coordinate system to obtain a second slope value includes the following steps:

[0102] Step S4423, obtaining the relative angle between the vehicle and the ground in the world coordinate system;

[0103] Step S4424: Convert the first slope value to the world coordinate system based on the relative angle to obtain a second slope value.

[0104] In the embodiment of the present invention, the relative angle can be understood as the tilt angle of the vehicle relative to the horizontal ground in the world coordinate system.

[0105] Obtaining the relative angle of the vehicle relative to the ground in the world coordinate system can be understood as the ARHUD accurately measuring the tilt angle of the vehicle relative to the horizontal ground, that is, the relative angle, through the vehicle's built-in sensors (such as gyroscopes and accelerometers).

[0106] Based on the relative angle, the first slope value is converted to the world coordinate system to obtain the second slope value. It can be understood that ARHUD uses the relative angle to convert the first slope value (indicating the ground inclination in the vehicle's driving direction) originally measured in the vehicle coordinate system to the world coordinate system to obtain the second slope value.

[0107] In an embodiment of the present invention, the first slope value measured in the vehicle coordinate system is converted to the world coordinate system through the relative angle. The obtained second slope value can more accurately reflect the ground inclination during the vehicle's driving process. It is also the basis for the ARHUD system to perform delay compensation and dynamic display content adjustment, ensuring a high degree of consistency between the display of virtual information and the actual movement state of the vehicle, thereby improving driving safety and experience.

[0108] Optionally, converting the first slope value into a world coordinate system based on the relative angle to obtain the second slope value includes the following steps:

[0109] The first slope value is converted to a world coordinate system based on the relative angle to obtain a third slope value;

[0110] The third slope value is screened to obtain a second slope value.

[0111] In the embodiment of the present invention, converting the first slope value to the world coordinate system based on the relative angle to obtain the third slope value can be understood as the ARHUD performing a coordinate system conversion on the first slope value measured in the vehicle coordinate system based on the relative angle of the vehicle in the world coordinate system, adjusting it to the world coordinate system, and obtaining the third slope value.

[0112] Filtering the third slope value to obtain the second slope value can be understood as after obtaining the converted third slope value, the ARHUD further performs data filtering and processing operations, the purpose of which is to filter out the valid slope information that best matches the current driving environment and conditions from the third slope value, thereby obtaining the second slope value. Exemplarily, the screening process may include removing outliers, applying a filtering algorithm to smooth data fluctuations, or dynamically adjusting the slope value based on the vehicle's speed and direction. By timely updating and eliminating information that is no longer valid or relevant, the system can ensure that it can continue to provide an augmented reality display that closely matches the vehicle's current state, thereby enhancing the driver's driving experience and safety. This is not limited here.

[0113] In this embodiment of the present invention, the ARHUD uses the vehicle's relative angle information to convert the first slope value obtained in the ego-vehicle coordinate system and adjust it to the world coordinate system, thereby obtaining a third slope value. This ensures accurate correspondence between slope information in different coordinate systems. The system then filters this third slope value to remove abnormal data, smooth data fluctuations, and adapt to the vehicle's dynamic changes, ultimately obtaining a second slope value. This ensures that the virtual information accurately reflects the actual conditions of the vehicle's travel path in real time, providing the driver with more intuitive and safe driving assistance information.

[0114] Optionally, determining the horizontal plane motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative time delay, and second slope value of the object to be prompted and the vehicle includes the following steps:

[0115] Obtain the first relative velocity between the object to be prompted and the vehicle in the vehicle coordinate system;

[0116] Convert the first relative velocity into the world coordinate system to obtain the second relative velocity;

[0117] Multiplying the second relative speed by the relative time delay to obtain a predicted motion vector of the object to be prompted in the world coordinate system;

[0118] The predicted motion vector is projected onto a horizontal plane based on the second slope value to obtain a horizontal plane motion vector.

[0119] In the embodiment of the present invention, the predicted motion vector can be understood as a vector representation of the estimated future position change of an object (such as a neighboring vehicle) based on the current speed and time delay of the object relative to the vehicle.

[0120] Obtaining the first relative speed between the object to be prompted and the vehicle in the vehicle coordinate system can be understood as follows: in the vehicle coordinate system, the ARHUD first measures and records the relative speed between the object to be prompted (such as the vehicle in front) and the vehicle, that is, the speed difference between the object to be prompted and the vehicle.

[0121] Converting the first relative velocity to the world coordinate system to obtain the second relative velocity can be understood as the ARHUD converting the first relative velocity in the vehicle coordinate system to the world coordinate system to obtain the second relative velocity.

[0122] Multiplying the second relative speed with the relative time delay to obtain the predicted motion vector of the object to be prompted in the world coordinate system can be understood as ARHUD using the second relative speed and the relative time delay from system perception to display to perform multiplication operation to calculate the expected displacement of the nearby vehicle within this time delay.

[0123] Projecting the predicted motion vector onto the horizontal plane based on the second slope value to obtain the horizontal plane motion vector can be understood as, taking into account the slope changes of the vehicle, ARHUD projects the predicted motion vector onto the horizontal plane to generate a horizontal plane motion vector.

[0124] In an embodiment of the present invention, ARHUD captures a first relative speed between the object to be prompted (such as a nearby vehicle) and the vehicle in the vehicle coordinate system, and obtains a second relative speed by converting the first relative speed to the world coordinate system. ARHUD multiplies the second relative speed by the relative time delay to generate a predicted motion vector. Taking into account the slope changes during driving, the system further projects the predicted motion vector to the horizontal plane based on the second slope value to generate a horizontal plane motion vector, ensuring the accurate correspondence between the display content of the auxiliary elements and the real-world dynamics, thereby improving the accuracy and reliability of driving assistance.

[0125] Optionally, converting the fourth display position into the real-time coordinate system of the vehicle based on the six-degree-of-freedom information of the vehicle to obtain the second display position includes the following steps:

[0126] Determine the real-time coordinate system of the ego vehicle based on the six-degree-of-freedom information of the ego vehicle;

[0127] The fourth display position is converted to the real-time vehicle coordinate system to obtain the second display position.

[0128] In the embodiments of the present invention, the ego-vehicle real-time coordinate system can be understood as a dynamic coordinate framework established based on the vehicle's current six-degree-of-freedom information. For example, the ego-vehicle real-time coordinate system can be continuously updated as the vehicle's driving state changes. Whenever the vehicle's position, orientation, or posture changes, the real-time coordinate system will be adjusted accordingly to maintain consistency with the vehicle's actual state, without limitation herein.

[0129] Determining the ego vehicle's real-time coordinate system based on its six degrees of freedom (6DOF) can be understood as constructing a coordinate system that fully matches the vehicle's current driving state using the 6DOF information (including 3D position and 3D attitude changes) collected by the vehicle's sensors (such as an inertial measurement unit and a global positioning system). This real-time coordinate system not only takes into account the vehicle's position (X, Y, Z) but also accurately reflects changes in its pitch, roll, and yaw angles. This ensures that regardless of the vehicle's dynamic conditions (straight driving, turning, uphill or downhill), the coordinate system reflects its attitude and position in real time, providing a dynamic and accurate reference framework for subsequent coordinate transformations and information display.

[0130] Converting the fourth display position to the real-time coordinate system of the vehicle to obtain the second display position can be understood as converting the fourth display position to a real-time coordinate system synchronized with the vehicle state to obtain display position information that better matches the vehicle's current driving environment, that is, the second display position.

[0131] In an embodiment of the present invention, by integrating the six-degree-of-freedom information of the vehicle, including the three-dimensional position and three-dimensional posture (pitch, roll, and yaw) of the vehicle, the ARHUD dynamically constructs a real-time coordinate system of the vehicle, and converts the preprocessed fourth display position into the real-time coordinate system of the vehicle, thereby obtaining the second display position, ensuring that the position information of the virtual content is completely adjusted according to the real-time status of the vehicle, so that the most accurate and fitting augmented reality information can be presented in the driver's current field of view, greatly improving the real-time and accuracy of driving assistance.

[0132] Figure 5 FIG. 1 is a schematic diagram of adjusting display content based on slope information according to an embodiment of the present invention. Figure 5As shown in the figure, first, ARHUD converts the perceived object in the ego vehicle coordinate system into an object in the world coordinate system as observation data, that is, by obtaining the perceived object in the ego vehicle coordinate system, and then obtaining the perceived object in the world coordinate system based on the ego vehicle's position information and heading angle in the world coordinate system. Secondly, ARHUD converts the corresponding slope value in the ego vehicle coordinate system into the slope value in the world coordinate system, that is, based on the relative angle, the slope value identified in the ego vehicle coordinate system is converted to the world coordinate system and filtered to determine the target slope value. Next, ARHUD performs three-dimensional dynamic delay compensation on dynamic obstacles (next car) based on the slope information in the world coordinate system, that is, multiplying the relative speed of the ego vehicle and the next car in the world coordinate system by the relative time delay to obtain a predicted motion vector, and then projecting the predicted motion vector onto the horizontal plane according to the slope to obtain a horizontal plane motion vector. Finally, ARHUD converts the position information of the adjacent vehicle in the world coordinate system into the real-time coordinate system of the vehicle based on the six degrees of freedom information of the vehicle itself, and obtains the coordinate information of the adjacent vehicle in the real-time coordinate system of the vehicle itself. In this way, it can present the most accurate and fitting augmented reality information in the driver's current field of view, greatly improving the real-time and accuracy of driving assistance, and providing the driver with a more intuitive and safe driving assistance experience.

[0133] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0134] According to an embodiment of the present invention, an embodiment of an augmented reality head-up display device for vehicle auxiliary element information is provided. It should be noted that the device can be used to execute the above-mentioned augmented reality head-up display method for vehicle auxiliary element information.

[0135] Figure 6 is a schematic diagram of an augmented reality head-up display device for vehicle auxiliary element information according to an embodiment of the present invention, Figure 6As shown, the augmented reality head-up display device 600 for vehicle auxiliary element information includes: an acquisition module 601, which is used to obtain the environmental perception data of the vehicle and the slope value corresponding to the vehicle in response to the vehicle being in the augmented reality auxiliary element prompt mode, wherein the change in the slope value is used to reflect the shaking state of the vehicle; a determination module 602, which is used to determine the auxiliary element information based on the environmental perception data, wherein the auxiliary element information includes a prompt image of the object to be prompted; a delay compensation module 603, which is used to perform delay compensation on the first display position of the auxiliary element information based on the slope value to obtain a second display position; and a display module 604, which is used to display the auxiliary element information at the second display position within the augmented reality head-up display field of view of the vehicle.

[0136] Furthermore, the delay compensation module 603 is also used to obtain a first display position of the object to be prompted in the vehicle coordinate system and a first slope value corresponding to the vehicle in the vehicle coordinate system; convert the first display position to the world coordinate system to obtain a third display position, and convert the first slope value to the world coordinate system to obtain a second slope value; determine the horizontal plane motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative delay and second slope value between the object to be prompted and the vehicle, wherein the horizontal plane motion vector is used to represent the motion direction of the object to be prompted and the distance offset of the object to be prompted, and the relative delay includes the dynamic delay of the prompt image of the object to be prompted during the acquisition, transmission and calculation process, as well as the fixed delay during the rendering and display process; determine the fourth display position of the object to be prompted in the world coordinate system based on the third display position and the horizontal plane motion vector; convert the fourth display position to the real-time coordinate system of the vehicle based on the six-degree-of-freedom information of the vehicle to obtain the second display position.

[0137] Furthermore, the time delay compensation module 603 is also used to obtain the vehicle position information and heading angle of the vehicle in the world coordinate system; based on the vehicle position information and heading angle, the first display position is converted to the world coordinate system to obtain a third display position.

[0138] Furthermore, the time delay compensation module 603 is further configured to obtain a relative angle between the vehicle and the ground in a world coordinate system; and convert the first slope value into the world coordinate system based on the relative angle to obtain a second slope value.

[0139] Furthermore, the time delay compensation module 603 is further configured to convert the first slope value into a world coordinate system based on the relative angle to obtain a third slope value; and filter the third slope value to obtain a second slope value.

[0140] Furthermore, the time delay compensation module 603 is also used to obtain a first relative speed between the object to be prompted and the vehicle in the vehicle coordinate system; convert the first relative speed to the world coordinate system to obtain a second relative speed; multiply the second relative speed by the relative delay to obtain a predicted motion vector of the object to be prompted in the world coordinate system; and project the predicted motion vector onto the horizontal plane based on the second slope value to obtain a horizontal plane motion vector.

[0141] Furthermore, the time delay compensation module 603 is further configured to determine the real-time coordinate system of the vehicle based on the six-degree-of-freedom information of the vehicle; and convert the fourth display position into the real-time coordinate system of the vehicle to obtain the second display position.

[0142] Furthermore, the display module 604 is also used to display that the objects to be prompted include static objects or dynamic objects, and the dynamic objects include at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian or an animal within the augmented reality head-up display field of view of the vehicle, and the static objects include at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian or an animal within the augmented reality head-up display field of view of the vehicle.

[0143] According to another aspect of an embodiment of the present invention, a vehicle is further provided, and the vehicle is used to execute the augmented reality head-up display method of vehicle auxiliary element information in various embodiments of the present invention.

[0144] According to another aspect of an embodiment of the present invention, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the augmented reality head-up display method of vehicle auxiliary element information in various embodiments of the present invention.

[0145] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored executable program, wherein when the executable program is running, the device where the computer-readable storage medium is located is controlled to execute the augmented reality head-up display method of vehicle auxiliary element information in various embodiments of the present invention.

[0146] According to another aspect of an embodiment of the present invention, a computer program product is further provided, including a computer program. When the computer program is executed by a processor, the computer program implements the augmented reality head-up display method of vehicle auxiliary element information in various embodiments of the present invention.

[0147] In the above embodiments of the present invention, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only exemplary. For example, the division of the units can be a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0149] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0150] In addition, the functional units in the various embodiments of the present invention may be integrated into a single processing unit, each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0151] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, magnetic disk or optical disk, etc. Various media that can store program codes.

[0152] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present invention. These improvements and modifications should also be regarded as within the scope of protection of the present invention.

Claims

1. An augmented reality head-up display method for vehicle auxiliary element information, characterized in that: include: In response to the ego vehicle being in the augmented reality auxiliary element prompt mode, obtaining environmental perception data of the ego vehicle and a slope value corresponding to the ego vehicle, wherein a change in the slope value is used to reflect a shaking state of the ego vehicle; Determining auxiliary element information based on the environmental perception data, wherein the auxiliary element information includes a prompt image of the object to be prompted; Performing time delay compensation on the first display position of the auxiliary element information based on the slope value to obtain a second display position; The auxiliary element information is displayed at the second display position within the augmented reality head-up display field of view of the vehicle.

2. The method according to claim 1, characterized in that The performing time delay compensation on the first display position of the auxiliary element information based on the slope value to obtain a second display position includes: Acquire the first display position of the object to be prompted in the vehicle coordinate system, and a first slope value corresponding to the vehicle in the vehicle coordinate system; Converting the first display position to a world coordinate system to obtain a third display position, and converting the first slope value to the world coordinate system to obtain a second slope value; Determining a horizontal plane motion vector of the object to be prompted in the world coordinate system based on the relative speed of the object to be prompted and the ego-vehicle, the relative time delay, and the second slope value, wherein the horizontal plane motion vector is used to represent the motion direction of the object to be prompted and the distance offset of the object to be prompted, and the relative time delay includes the dynamic time delay of the prompt image of the object to be prompted during the acquisition, transmission, and calculation processes, as well as the fixed time delay during the rendering and display process; determining a fourth display position of the object to be prompted in the world coordinate system based on the third display position and the horizontal plane motion vector; The fourth display position is converted into the real-time coordinate system of the vehicle based on the six-degree-of-freedom information of the vehicle to obtain the second display position.

3. The method according to claim 2, characterized in that The converting the first display position into a world coordinate system to obtain a third display position includes: Obtaining the vehicle's position information and heading angle in the world coordinate system; The first display position is converted into the world coordinate system based on the vehicle position information and the heading angle to obtain the third display position.

4. The method according to claim 2, characterized in that The converting the first slope value into the world coordinate system to obtain a second slope value includes: Obtaining a relative angle between the vehicle and the ground in the world coordinate system; The first slope value is converted to the world coordinate system based on the relative angle to obtain the second slope value.

5. The method according to claim 4, characterized in that The converting the first slope value into the world coordinate system based on the relative angle to obtain the second slope value includes: Converting the first slope value to the world coordinate system based on the relative angle to obtain a third slope value; The third slope value is screened to obtain the second slope value.

6. The method according to claim 2, characterized in that The determining of the horizontal plane motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative time delay and the second slope value between the object to be prompted and the vehicle includes: Acquire a first relative speed between the object to be prompted and the vehicle in the vehicle coordinate system; Converting the first relative velocity into the world coordinate system to obtain a second relative velocity; Multiplying the second relative speed by the relative time delay to obtain a predicted motion vector of the object to be prompted in the world coordinate system; The predicted motion vector is projected onto a horizontal plane based on the second slope value to obtain the horizontal plane motion vector.

7. The method according to claim 2, characterized in that The converting the fourth display position into the real-time coordinate system of the vehicle based on the six-degree-of-freedom information of the vehicle to obtain the second display position includes: Determining the real-time coordinate system of the vehicle based on the six-degree-of-freedom information of the vehicle; The fourth display position is converted into the vehicle real-time coordinate system to obtain the second display position.

8. The method according to any one of claims 1 to 7, characterized in that The objects to be prompted include static objects or dynamic objects, and the dynamic objects include at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian or an animal within the field of view of the augmented reality head-up display of the vehicle; and the static objects include at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian or an animal within the field of view of the augmented reality head-up display of the vehicle.

9. An augmented reality head-up display device for vehicle auxiliary element information, characterized in that: include: an acquisition module, configured to acquire, in response to the vehicle being in an augmented reality auxiliary element prompt mode, environmental perception data of the vehicle and a slope value corresponding to the vehicle, wherein a change in the slope value is used to reflect a shaking state of the vehicle; a determination module, configured to determine auxiliary element information based on the environmental perception data, wherein the auxiliary element information includes a prompt image of an object to be prompted; a time delay compensation module, configured to perform time delay compensation on the first display position of the auxiliary element information based on the slope value to obtain a second display position; A display module is configured to display the auxiliary element information at the second display position within the augmented reality head-up display field of view of the vehicle.

10. A vehicle, characterized in that: The vehicle is used to execute the augmented reality head-up display method of vehicle auxiliary element information as described in any one of claims 1 to 8.

11. An electronic device comprising a memory and a processor, characterized in that: The memory stores a computer program, and the processor is configured to run the computer program to execute the augmented reality head-up display method for vehicle auxiliary element information as claimed in any one of claims 1 to 8.

Images