Augmented reality heads-up display method, device and electronic device for vehicle auxiliary element information

Abstract

The application discloses an augmented reality head-up display method and device for vehicle auxiliary element information and an electronic device. The method comprises the following steps: in response to the fact that the ego vehicle is in an augmented reality auxiliary element prompt mode, acquiring environment perception data of the ego vehicle and a slope value corresponding to the ego vehicle, wherein the change amount of the slope value is used to reflect the shaking state of the ego vehicle; determining auxiliary element information based on the environment perception data, wherein the auxiliary element information comprises a prompt image of a to-be-prompted object; performing time delay compensation on a first display position of the auxiliary element information based on the slope value to obtain a second display position; and displaying the auxiliary element information at the second display position within the field of view of the augmented reality head-up display of the ego vehicle. The application solves the technical problem of visual misplacement caused by the delay when information is displayed by using an augmented reality head-up display in the related art.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and more specifically, to an augmented reality head-up display method, apparatus, and electronic device for displaying vehicle assistance element information. Background Technology

[0002] To provide users with information about the road conditions around their vehicle while driving, Simulation Reality (SR) technology, combined with information displayed on the in-vehicle infotainment screen, is typically used to provide driving guidance. The bird's-eye view provided by SR mode differs significantly from the driver's actual first-person perspective. In actual driving, the driver operates from an in-vehicle first-person perspective, while the global view of SR mode often makes users feel disconnected from reality, reducing the intuitiveness and immersiveness of the driving assistance information. Furthermore, this perspective shift may cause discomfort for some users, affecting the user experience. While the road condition information displayed on the in-vehicle infotainment screen is relatively accurate, the screen is limited by a fixed viewing angle and limited display space, making it unable to provide an immersive experience that matches the driver's real-time line of sight.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This invention provides an augmented reality head-up display method, apparatus, and electronic device for displaying vehicle auxiliary element information, in order to at least solve the technical problem of visual misalignment caused by delay when displaying information using an augmented reality head-up display in related technologies.

[0005] According to one aspect of the present invention, an augmented reality head-up display method for vehicle auxiliary element information is provided, comprising: in response to the vehicle being in an augmented reality auxiliary element prompting mode, acquiring environmental perception data of the vehicle and a slope value corresponding to the vehicle, wherein the change in the slope value is used to reflect the shaking state of the vehicle; determining auxiliary element information based on the environmental perception data, wherein the auxiliary element information includes a prompting image of the object to be prompted; performing time delay compensation on a first display position of the auxiliary element information based on the slope value to obtain a second display position; and displaying the auxiliary element information at the second display position within the augmented reality head-up display field of view of the vehicle.

[0006] Further, the second display position is obtained by performing time delay compensation on the first display position of the auxiliary element information based on the slope value, including: acquiring the first display position of the object to be prompted in the vehicle coordinate system, and the first slope value corresponding to the vehicle in the vehicle coordinate system; transforming the first display position to the world coordinate system to obtain the third display position, and transforming the first slope value to the world coordinate system to obtain the second slope value; determining the horizontal plane motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative time delay, and second slope value between the object to be prompted and the vehicle, wherein the horizontal plane motion vector is used to represent the motion direction of the object to be prompted and the distance offset of the object to be prompted, and the relative time delay includes the dynamic time delay of the prompt image of the object to be prompted during the acquisition, transmission, and calculation process, as well as the fixed time delay during the rendering and display process; determining the fourth display position of the object to be prompted in the world coordinate system based on the third display position and the horizontal plane motion vector; and transforming the fourth display position to the vehicle real-time coordinate system based on the six degrees of freedom information of the vehicle to obtain the second display position.

[0007] Furthermore, transforming the first display position to the world coordinate system to obtain the third display position includes: acquiring the vehicle's position information and heading angle in the world coordinate system; and transforming the first display position to the world coordinate system based on the vehicle's position information and heading angle to obtain the third display position.

[0008] Furthermore, converting the first slope value to the world coordinate system to obtain the second slope value includes: obtaining the relative angle between the vehicle and the ground in the world coordinate system; and converting the first slope value to the world coordinate system based on the relative angle to obtain the second slope value.

[0009] Furthermore, the process of converting the first slope value to the world coordinate system based on the relative angle to obtain the second slope value includes: converting the first slope value to the world coordinate system based on the relative angle to obtain the third slope value; and filtering the third slope value to obtain the second slope value.

[0010] Furthermore, determining the horizontal motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative time delay, and second slope value between the object to be prompted and the vehicle includes: obtaining the first relative speed between the object to be prompted and the vehicle in the vehicle coordinate system; transforming the first relative speed to the world coordinate system to obtain the second relative speed; multiplying the second relative speed by the relative time delay to obtain the predicted motion vector of the object to be prompted in the world coordinate system; and projecting the predicted motion vector onto the horizontal plane based on the second slope value to obtain the horizontal motion vector.

[0011] Furthermore, transforming the fourth display position to the vehicle's real-time coordinate system based on the vehicle's six degrees of freedom information to obtain the second display position includes: determining the vehicle's real-time coordinate system based on the vehicle's six degrees of freedom information; and transforming the fourth display position to the vehicle's real-time coordinate system to obtain the second display position.

[0012] Furthermore, the object to be prompted includes static objects or dynamic objects. Dynamic objects include at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian, or an animal within the field of view of the augmented reality head-up display of the vehicle. Static objects include at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian, or an animal within the field of view of the augmented reality head-up display of the vehicle.

[0013] According to another aspect of the present invention, an augmented reality head-up display device for vehicle auxiliary element information is also provided, comprising: an acquisition module, configured to acquire environmental perception data of the vehicle and a slope value corresponding to the vehicle in response to the vehicle being in an augmented reality auxiliary element prompting mode, wherein the change in the slope value is used to reflect the shaking state of the vehicle; a determination module, configured to determine auxiliary element information based on the environmental perception data, wherein the auxiliary element information includes a prompting image of the object to be prompted; a time delay compensation module, configured to perform time delay compensation on a first display position of the auxiliary element information based on the slope value to obtain a second display position; and a display module, configured to display the auxiliary element information at the second display position within the augmented reality head-up display field of view of the vehicle.

[0014] Furthermore, the latency compensation module is also used to obtain the first display position of the object to be prompted in the vehicle coordinate system, and the first slope value of the vehicle in the vehicle coordinate system; to transform the first display position to the world coordinate system to obtain the third display position, and to transform the first slope value to the world coordinate system to obtain the second slope value; to determine the horizontal motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative latency, and second slope value between the object to be prompted and the vehicle, wherein the horizontal motion vector is used to represent the motion direction of the object to be prompted and the distance offset of the object to be prompted, and the relative latency includes the dynamic latency of the prompt image of the object to be prompted during the acquisition, transmission, and calculation process, as well as the fixed latency during the rendering and display process; to determine the fourth display position of the object to be prompted in the world coordinate system based on the third display position and the horizontal motion vector; and to transform the fourth display position to the vehicle real-time coordinate system based on the six degrees of freedom information of the vehicle to obtain the second display position.

[0015] Furthermore, the time delay compensation module is also used to obtain the vehicle's position information and heading angle in the world coordinate system; based on the vehicle's position information and heading angle, the first display position is transformed to the world coordinate system to obtain the third display position.

[0016] Furthermore, the time delay compensation module is also used to obtain the relative angle between the vehicle and the ground in the world coordinate system; based on the relative angle, the first slope value is converted to the world coordinate system to obtain the second slope value.

[0017] Furthermore, the time delay compensation module is also used to transform the first slope value to the world coordinate system based on the relative angle to obtain the third slope value; and to filter the third slope value to obtain the second slope value.

[0018] Furthermore, the time delay compensation module is also used to obtain the first relative velocity between the object to be prompted and the vehicle in the vehicle coordinate system; to transform the first relative velocity to the world coordinate system to obtain the second relative velocity; to multiply the second relative velocity by the relative time delay to obtain the predicted motion vector of the object to be prompted in the world coordinate system; and to project the predicted motion vector onto the horizontal plane based on the second slope value to obtain the horizontal plane motion vector.

[0019] Furthermore, the time delay compensation module is also used to determine the vehicle's real-time coordinate system based on the vehicle's six degrees of freedom information; and to transform the fourth display position to the vehicle's real-time coordinate system to obtain the second display position.

[0020] Furthermore, the display module is also used to display the object to be prompted, including static objects or dynamic objects. Dynamic objects include at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian, or an animal within the field of view of the augmented reality head-up display of the vehicle. Static objects include at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian, or an animal within the field of view of the augmented reality head-up display of the vehicle.

[0021] According to another aspect of the present invention, a vehicle is also provided, which is used to perform the augmented reality head-up display method for vehicle auxiliary element information in various embodiments of the present invention.

[0022] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform an augmented reality head-up display method for vehicle auxiliary element information according to various embodiments of the present invention.

[0023] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the augmented reality head-up display method for vehicle auxiliary element information in various embodiments of the present invention.

[0024] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the augmented reality head-up display method for vehicle auxiliary element information in various embodiments of the present invention.

[0025] In this embodiment of the invention, a combination of dynamic slope compensation and time delay correction is employed. The change in slope value is precisely calculated to reflect the vehicle's vibration state, and auxiliary element information, including the prompt image of the object to be prompted, is determined based on environmental perception data. Real-time time delay compensation is applied to the first display position of the auxiliary element information based on the slope value, resulting in a more accurate second display position. The auxiliary element information is displayed at the second display position within the vehicle's ARHUD field of view, ensuring the immediacy and accuracy of the information. This reduces ARHUD display latency and improves the matching degree between the displayed content and the actual environment, thereby significantly enhancing driving safety and user experience. Furthermore, it solves the technical problem of visual misalignment caused by latency when displaying information using augmented reality head-up displays in related technologies. Attached Figure Description

[0026] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0027] Figure 1 This is a schematic diagram of a self-driving vehicle and a vehicle alongside it, based on existing technology and a simulated reality.

[0028] Figure 2 This is a schematic diagram illustrating the display of road driving information on a vehicle infotainment screen based on existing technology.

[0029] Figure 3 This is a schematic diagram of an augmented reality head-up display for displaying information about the surroundings of a vehicle, according to an embodiment of the present invention.

[0030] Figure 4 This is a flowchart illustrating an augmented reality head-up display method for vehicle auxiliary element information according to an embodiment of the present invention;

[0031] Figure 5 This is a schematic diagram illustrating the adjustment of displayed content based on slope information according to an embodiment of the present invention;

[0032] Figure 6 This is a schematic diagram of an augmented reality head-up display device for vehicle auxiliary element information according to an embodiment of the present invention. Detailed Implementation

[0033] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0034] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0035] Figure 1 This is a schematic diagram of a self-driving vehicle and a vehicle alongside it, based on existing technology and a simulated reality, such as... Figure 1 As shown, the vehicle and the vehicle next to it are displayed in the same frame, and the relative relationship between the vehicle and the vehicle next to it is displayed correctly. However, the bird's-eye view provided by SR mode is significantly different from the driver's actual first-person view. Therefore, the user's sense of motion is not strong. In addition, the relative position and motion state between vehicles are generally correct in SR mode, but due to the lack of accurate tracking of the real-time dynamics of individual vehicles, especially in high-speed driving or emergency situations, this bird's-eye view may not provide enough detailed information. Furthermore, this perspective change may make some users feel uncomfortable and affect the user experience. Figure 2 This is a schematic diagram illustrating the display of road driving information on a vehicle's infotainment screen, based on existing technology. Figure 2 As shown, although the road conditions around the vehicle displayed on the in-vehicle screen are relatively accurate, the screen may display multiple pieces of information at the same time, such as navigation instructions, entertainment system controls, and vehicle status. Too much or too dense information can sometimes make it difficult for the driver to quickly filter out key information, thus increasing the cognitive burden.

[0036] To address the problems of the aforementioned simulation technology and in-vehicle infotainment screens, Augmented Reality Head-Up Display (ARHUD) has been proposed as an emerging head-up display technology. It is mainly used to guide users to identify and judge path information such as the road ahead, obstacles, and traffic signals. Figure 3 This is a schematic diagram illustrating the display of information surrounding a vehicle based on an augmented reality head-up display according to an embodiment of the present invention, such as... Figure 3 As shown, ARHUD is an innovative driver assistance technology that overlays virtual information (such as navigation instructions, vehicle status, road condition warnings, weather information, etc.) onto the real-world scene directly in front of the driver via the windshield or a special transparent display. ARHUD's key advantage lies in its ability to seamlessly integrate virtual information with the driver's real line of sight, providing intuitive and safe driving assistance while reducing driver distraction. However, a major challenge in using ARHUD to provide driving assistance is latency compensation—that is, ensuring that the displayed information remains synchronized with the dynamics of the real world and avoiding visual misalignment caused by delay.

[0037] While both displaying navigation information on the vehicle's infotainment screen and displaying navigation information via an AR HUD can provide navigation guidance, displaying navigation information via an AR HUD is not simply changing the subject of the same navigation information. Rather, there are fundamental differences in the logic and effects of generating and implementing the displayed navigation information. These differences determine their respective characteristics and application scenarios.

[0038] The core of AR HUD lies in integrating the navigation guidance light carpet with the actual road, merging perceived lane lines, surrounding vehicles, non-motorized vehicles, pedestrians, and animals with the real environment, and presenting it on the windshield in front of the driver's line of sight. In effect, this alignment and integration of virtual navigation information with the real world ensures a high degree of consistency between virtual and real-world elements, providing users with an intuitive and immersive driving assistance experience.

[0039] In contrast, the navigation information displayed on the in-vehicle infotainment screen is presented as virtual information such as navigation routes (not navigation guide light carpets) and the vehicle itself and surrounding vehicles. These are rendered through animations on a fixed display screen inside the car. The navigation routes and virtual information do not need to be spatially integrated with the external environment. Their main purpose is to show the user what the surrounding environment is like, and they do not need to be as precise.

[0040] The specific differences are as follows:

[0041] Table 1

[0042]

[0043] As shown in Table 1, regarding whether virtual information aligns with actual roads, the fundamental difference between AR HUD and traditional in-vehicle screens in navigation display function lies in their ability to overlay navigation information directly within the driver's field of vision due to their augmented reality characteristics. AR HUDs can display a navigation guidance light carpet on the current road, aligning it perfectly with the actual road. This allows users to clearly understand how to drive ahead, providing a very intuitive experience. In contrast, in-vehicle screens do not require direct visual integration with the road and do not display navigation information via a light carpet. Instead, they use color-coded markers to indicate which roads are passable, letting the user know which roads are available, and ultimately, the user decides which one to take. In addition, AR HUD can display lane lines ahead within a certain range, as well as information about motor vehicles, non-motor vehicles, pedestrians, or animals in the surrounding environment. It can intuitively alert users to external factors that may affect driving operations. Especially in environments with low visibility, where users may not notice their surroundings, this information can greatly help users make decisions to avoid traffic accidents. However, the navigation on the car's screen will render some surrounding objects, but it can only give a general idea that there may be an object nearby. Users cannot intuitively perceive the specific location of the object.

[0044] Regarding whether real-time prediction of the vehicle's position is necessary, since the underlying implementation logic of these two technologies is different, for example, suppose that in low visibility conditions, we need to use an ARHUD to display the position element of the vehicle in front to help users identify the distance between their own vehicle and the vehicle in front. How to ensure that this position element fits the vehicle in front is very important for users. It is necessary to predict the position of the vehicle in front to achieve a better fit. For example, if the vehicle in front suddenly accelerates or decelerates, in order for the animation rendering on the ARHUD to keep up with the vehicle in front, it is necessary to predict the position of the vehicle in front. Specifically, we need to first obtain multiple predicted positions of the vehicle in front (for example, some predicted positions of the vehicle in front are after acceleration, and some predicted positions of the vehicle in front are after deceleration). Then, we need to average the positions of the vehicles in front in the current frame to obtain a predicted position. This predicted position can then be used to display on the ARHUD. In this case, even if the vehicle in front suddenly accelerates, this predicted position will still be relatively close to the vehicle in front.

[0045] The navigation screen displayed on the car's infotainment system will show an animated rendering of the car in front. However, this animation rendering does not have a strong requirement to be perfectly aligned with the car in front. Even if there is a certain distance difference between the rendered position of the car in front and its actual position, it does not matter. The car's infotainment system only needs to indicate a rough relative position of the car in front, and does not need to be particularly accurate. In other words, the navigation displayed on the car's infotainment system does not predict the speed of the car in front. Instead, it uses perception data to detect and calculate in real time whether the car will collide with the car in front at present, without needing to know whether the car will collide with the car in front in the future.

[0046] Regarding the issue of the navigation guide light carpet exceeding the screen width, since the ARHUD only displays a certain range of content in front of the vehicle, and the navigation guide light carpet must be adapted to road conditions, such as when turning right or making a U-turn, the navigation guide light carpet may exceed the screen width. However, the screen displays the passable road ahead (without showing the navigation guide light carpet), so even if a right turn or U-turn is required, the passable road ahead will be shown in the image. Therefore, the screen does not have the issue of the navigation guide light carpet exceeding the screen width.

[0047] According to an embodiment of the present invention, an embodiment of an augmented reality head-up display method for vehicle auxiliary element information is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0048] Figure 4 This is a flowchart illustrating an augmented reality head-up display method for vehicle auxiliary element information according to an embodiment of the present invention, as shown below. Figure 4 As shown, the method includes the following steps:

[0049] Step S40: In response to the vehicle being in augmented reality assistive element prompting mode, acquire the vehicle's environmental perception data and the corresponding slope value, wherein the change in slope value is used to reflect the vehicle's shaking state.

[0050] In this embodiment of the invention, the Augmented Reality Assistive Element Prompt Mode can be understood as a working mode of a vehicle infotainment system or driver assistance system, which uses ARHUD to directly display key driver assistance information in the form of virtual images within the user's actual field of vision, typically through the windshield or other transparent display screen. The Augmented Reality Assistive Element Prompt Mode can provide assistance information closely integrated with actual driving conditions, helping users better understand and cope with the road environment; however, this is not limited to this specific mode.

[0051] Environmental perception data can be understood as information collected by various sensors of a vehicle (such as cameras, radar, lidar, etc.) to detect and analyze the environment around the vehicle, including but not limited to: the position, speed and acceleration of other vehicles, road conditions such as lane lines and traffic signs, obstacles such as pedestrians and bicycles, as well as weather conditions and visibility, which are not limited here.

[0052] The slope value corresponding to the vehicle can be understood as the incline of the road surface where the vehicle is located, usually expressed in degrees or percentages. The slope value corresponding to the vehicle reflects the degree of inclination of the road surface where the vehicle is currently located. The slope value is particularly important in the application of ARHUD because changes in slope affect the stability of the vehicle, which in turn affects the correct display position of auxiliary elements. No restrictions are imposed here.

[0053] The vibration of a vehicle can be understood as a change in vehicle stability caused by uneven road surfaces, vehicle vibration, or other factors. For example, during driving, a vehicle may experience vibration or shaking due to potholes, slopes, or wind resistance, which will cause changes in the slope value. This will directly affect the position of auxiliary elements displayed in the ARHUD, potentially leading to discrepancies between the displayed content and the actual environment, affecting driving safety and experience. However, this is not a limitation here.

[0054] In response to the vehicle being in augmented reality assist element prompt mode, acquiring the vehicle's environmental perception data and the corresponding slope value can be understood as the vehicle system starting to collect and process environmental perception data and the corresponding slope value when the vehicle's ARHUD assisted driving mode is activated.

[0055] The change in slope value is used to reflect the vibration state of the vehicle. It can be understood that when the vehicle encounters a road surface with different slopes during driving, the vehicle will react differently, resulting in different changes in the vehicle's posture. There are no restrictions on this.

[0056] In this embodiment of the invention, when the vehicle is operating in augmented reality assistive element prompting mode, the system actively collects environmental perception data and the current slope value, paying particular attention to the change in the slope value, as an indicator to assess the vehicle's vibration state. Through real-time analysis and processing of environmental and slope information, the system can more accurately adjust the display position of assistive elements on the ARHUD, ensuring that even when the vehicle vibrates or travels on undulating roads, the virtual information accurately corresponds to the real-world situation, greatly improving the practicality and safety of the ARHUD.

[0057] Step S42: Determine auxiliary element information based on environmental perception data, wherein the auxiliary element information includes the prompt image of the object to be prompted.

[0058] In this embodiment of the invention, auxiliary element information can be understood as key visual auxiliary information provided to the user. Exemplarily, auxiliary element information may include navigation instructions (such as turn arrows), hazard warnings (such as collision warnings), vehicle status prompts (such as displaying vehicle speed), road condition information (such as slippery road surface), and dynamic obstacle tracking (such as real-time tracking and marking of surrounding vehicles or pedestrians), without limitation.

[0059] The object to be prompted can be understood to include static objects or dynamic objects. Dynamic objects include at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian or animal within the field of view of the augmented reality head-up display of the vehicle. Static objects include at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian or animal within the field of view of the augmented reality head-up display of the vehicle, without any limitation.

[0060] The prompt image of the object to be prompted can be understood as a visual representation of auxiliary element information. It is a virtual image that the user sees in the ARHUD view, used to highlight or prompt important elements in the road environment, and there are no restrictions here.

[0061] Based on environmental perception data, auxiliary element information is determined. This auxiliary element information includes the prompt image of the object to be prompted. It can be understood that the ARHUD system can intelligently analyze the surrounding environment and generate corresponding auxiliary display information accordingly. The auxiliary information will be superimposed on the driver's actual field of vision in the form of visual images, seamlessly integrated with the real environment, and provide direct and effective driving assistance. No restrictions are imposed here.

[0062] In this embodiment of the invention, the ARHUD system uses data collected by vehicle sensors to generate targeted virtual assistance information, which is presented to the driver in an intuitive form. This not only improves the driving experience but also effectively reduces the driver's cognitive burden, allowing them to focus more on driving itself and improve their perception of the surrounding environment.

[0063] Step S44: Based on the slope value, perform time delay compensation on the first display position of the auxiliary element information to obtain the second display position.

[0064] In this embodiment of the invention, the first display position can be understood as the initial display position calculated by the ARHUD based on environmental perception data and slope value information when it receives such data. For example, due to delays in data processing and display updates, the first display position often deviates somewhat from the actual position, but this is not a limitation here.

[0065] Latency compensation can be understood as a crucial step in ARHUD technology, used to address the delay in matching displayed content with the position of objects in the real world. For example, the specific process of latency compensation might include: predicting vehicle motion: based on the vehicle's current speed, acceleration, and gradient, predicting the vehicle's position and attitude changes during the display delay time; adjusting the display position: considering the predicted displacement and attitude changes, adjusting the display position of auxiliary element information to ensure that, in the final display, the auxiliary element information accurately corresponds to the position of the object or event to be prompted in the real world; this is not limited to specific steps.

[0066] The second display position can be understood as the final display position of the auxiliary element information on the display device after time delay compensation, which is not restricted here.

[0067] The second display position is obtained by performing time delay compensation on the first display position of the auxiliary element information based on the slope value. This can be understood as follows: In the ARHUD system, the initial display position of the auxiliary element information on the display screen is adjusted by dynamically analyzing the slope changes of the vehicle driving, thereby obtaining the final position of the auxiliary element information on the display screen. There are no restrictions on this.

[0068] In this embodiment of the invention, the initial display position of the auxiliary element information is adjusted based on the slope value to ensure that the display position of the auxiliary element information is consistent with reality, thereby improving the accuracy and reliability of driving assistance and playing a significant role in enhancing driving safety.

[0069] Step S46: Display auxiliary element information at the second display position within the field of view of the vehicle's augmented reality head-up display.

[0070] In this embodiment of the invention, the augmented reality head-up display (ARHUD) field of view can be understood as the portion of the real-world environment that the driver can clearly see through the ARHUD, which is superimposed with virtual information. For example, the ARHUD field of view can be set in the lower area of ​​the vehicle's windshield, or displayed through a specially designed small transparent panel (usually located above the dashboard) to provide more intuitive and practical driving assistance; this is not limited to this specific example.

[0071] Displaying auxiliary element information at the second display position within the augmented reality head-up display's field of view of the vehicle can be understood as ARHUD displaying processed auxiliary information that takes into account the vehicle's motion state (especially slope changes) at the optimal position and time within the augmented reality head-up display's field of view. This ensures that the auxiliary element information seen by the driver within the field of view is real-time, accurate, and has direct reference value, thereby improving the driving experience and reducing the risk of accidents. No restrictions are imposed here.

[0072] In this embodiment of the invention, ARHUD displays auxiliary element information within the field of view of the vehicle's augmented reality head-up display. It not only takes into account the processing latency of onboard sensor data, but also focuses on the vehicle's motion characteristics under different road conditions, such as dynamic changes when going uphill or downhill. Through accurate prediction and matching, it ensures that the display of virtual information is highly synchronized with the real environment, providing the driver with a safer, more intuitive and effective driving assistance method.

[0073] In this embodiment of the invention, a combination of dynamic slope compensation and time delay correction is employed. The change in slope value is precisely calculated to reflect the vehicle's vibration state, and auxiliary element information, including the prompt image of the object to be prompted, is determined based on environmental perception data. Real-time time delay compensation is applied to the first display position of the auxiliary element information based on the slope value, resulting in a more accurate second display position. The auxiliary element information is displayed at the second display position within the vehicle's ARHUD field of view, ensuring the immediacy and accuracy of the information. This reduces ARHUD display latency and improves the matching degree between the displayed content and the actual environment, thereby significantly enhancing driving safety and user experience. Furthermore, it solves the technical problem of visual misalignment caused by latency when displaying information using augmented reality head-up displays in related technologies.

[0074] Optionally, in step S44, performing time delay compensation on the first display position of the auxiliary element information based on the slope value to obtain the second display position includes the following steps:

[0075] Step S441: Obtain the first display position of the object to be prompted in the vehicle coordinate system, and the first slope value of the vehicle in the vehicle coordinate system.

[0076] Step S442: Convert the first display position to the world coordinate system to obtain the third display position, and convert the first slope value to the world coordinate system to obtain the second slope value;

[0077] Step S443: Determine the horizontal motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative time delay and second slope value between the object to be prompted and the vehicle. The horizontal motion vector is used to represent the direction of motion of the object to be prompted and the distance offset of the object to be prompted. The relative time delay includes the dynamic time delay of the prompt image of the object to be prompted during the acquisition, transmission and calculation process, as well as the fixed time delay during the rendering and display process.

[0078] Step S444: Determine the fourth display position of the object to be prompted in the world coordinate system based on the third display position and the horizontal plane motion vector;

[0079] Step S445: Based on the six degrees of freedom information of the vehicle, the fourth display position is transformed into the real-time coordinate system of the vehicle to obtain the second display position.

[0080] In this embodiment of the invention, the vehicle coordinate system can be understood as a local coordinate system established with the vehicle as the center. In the vehicle coordinate system, the vehicle's position is regarded as the origin, and it is usually used to represent the position information of objects in the surrounding environment relative to the vehicle, but this is not limited here.

[0081] The first gradient value can be understood as a measurement or estimate in the vehicle's coordinate system, reflecting the current slope of the road where the vehicle is located. The gradient value is crucial to the display accuracy of ARHUD, especially when the vehicle is going uphill or downhill, as changes in gradient will affect the vehicle's attitude and dynamic behavior, which will not be limited here.

[0082] The world coordinate system can be understood as a global coordinate system used to describe absolute position information. Compared with the vehicle coordinate system, the world coordinate system is not affected by the vehicle's own motion, providing a stable reference frame, which is beneficial for more accurate position calculations and transformations. No restrictions are imposed here.

[0083] The second slope value can be understood as the transformation result of the first slope value in the world coordinate system. Transforming the slope value from the vehicle coordinate system to the world coordinate system helps to more accurately calculate the displacement and attitude changes of the object to be prompted in different coordinate systems, which is not restricted here.

[0084] Relative velocity can be understood as the velocity of the object to be prompted relative to the vehicle itself. In the ARHUD scenario, relative velocity is used to predict the expected position of the object to be prompted after a certain period of time, and there are no restrictions on this.

[0085] The horizontal motion vector can be understood as a calculation derived from relative velocity and relative time delay (including dynamic and fixed time delay), used to represent the horizontal trajectory of the object to be prompted, ignoring vertical changes (such as uphill or downhill). This vector contains the object's direction of motion and the expected horizontal displacement within the time interval, facilitating position prediction on a two-dimensional plane, which is not restricted here.

[0086] Six degrees of freedom (DOF) information can be understood as information describing the vehicle's motion state in three-dimensional space, including three translational degrees of freedom (movement along the x, y, and z axes) and three rotational degrees of freedom (rotation about the x, y, and z axes). In ARHUD technology, six-DOF information is used to update the vehicle's position and attitude in real time, ensuring accurate alignment between virtual information and the real world; however, this is not a limitation here.

[0087] Obtaining the first display position of the object to be prompted in the vehicle's coordinate system, and the corresponding first slope value of the vehicle in the same coordinate system, can be understood as follows: ARHUD uses onboard sensors and autonomous driving technology to collect and process data from the surrounding environment in real time to determine the relative position of the object to be prompted (such as vehicles, pedestrians, obstacles, etc.) in the vehicle's coordinate system, i.e., the first display position. Simultaneously, the system also detects and records the current slope value of the vehicle's movement, i.e., the first slope value.

[0088] Transforming the first display position to the world coordinate system to obtain the third display position, and transforming the first slope value to the world coordinate system to obtain the second slope value, can be understood as follows: in order to make more accurate predictions and position adjustments, ARHUD transforms the first display position and the first slope value in the vehicle coordinate system to a fixed world coordinate system.

[0089] Determining the horizontal motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative delay, and second slope value of the object to be prompted and the vehicle can be understood as follows: ARHUD considers multiple dynamic factors, including the relative speed of the object to be prompted and the vehicle, the relative delay required to process the displayed information (the sum of dynamic delay and fixed delay), and the second slope value, to calculate the horizontal motion vector of the object to be prompted in the world coordinate system.

[0090] Determining the fourth display position of the object to be prompted in the world coordinate system based on the third display position and the horizontal plane motion vector can be understood as follows: after obtaining the horizontal plane motion vector, the system predicts the exact position that the object to be prompted should be displayed in the world coordinate system after overcoming the display delay, i.e., the fourth display position.

[0091] Based on the vehicle's six degrees of freedom information, the fourth display position is transformed into the vehicle's real-time coordinate system to obtain the second display position. This can be understood as follows: in order to transform the predicted fourth display position into a form that the driver can intuitively see, the system uses the vehicle's six degrees of freedom information (including position, attitude, and dynamic changes) to transform the fourth display position from the world coordinate system back to the vehicle's real-time coordinate system.

[0092] In this embodiment of the invention, the ARHUD acquires the initial position information of the object to be prompted around the vehicle in the vehicle coordinate system and the vehicle's current first slope value. The above information is converted to the world coordinate system to obtain more accurate absolute position and slope information. Using the relative speed and relative time delay of the object to be prompted, combined with the second slope value, the horizontal motion vector of the object in the world coordinate system is calculated. Based on the third display position (i.e., the initial position of the object to be prompted in the world coordinate system) and the horizontal motion vector, the fourth display position of the object to be prompted after experiencing a relative time delay is determined. The fourth display position is converted back to the vehicle's real-time coordinate system to obtain the final second display position. This ensures that the position of the virtual information displayed on the ARHUD can highly match the actual position of the object in the driver's forward field of vision, providing accurate and timely driving assistance.

[0093] Optionally, in step S442, transforming the first display position to the world coordinate system to obtain the third display position includes the following steps:

[0094] Step S4421: Obtain the vehicle's position information and heading angle in the world coordinate system;

[0095] Step S4422: Based on the vehicle's position information and heading angle, the first display position is converted to the world coordinate system to obtain the third display position.

[0096] In this embodiment of the invention, the vehicle position information can be understood as the exact position coordinates of the vehicle in the world coordinate system.

[0097] The heading angle can be understood as the angular difference between the vehicle's current direction of travel and a fixed reference direction (such as north) in the world coordinate system, and is used to describe the vehicle's orientation or attitude.

[0098] Obtaining the vehicle's position and heading angle in the world coordinate system can be understood as follows: ARHUD obtains the vehicle's precise geographical location information, including its three-dimensional coordinates (longitude, latitude, and altitude) in the world coordinate system, as well as its current heading angle, through BeiDou or other satellite positioning technologies and the vehicle's own sensors (such as inertial measurement units).

[0099] The transformation of the first display position to the world coordinate system based on the vehicle's position information and heading angle to obtain the third display position can be understood as follows: ARHUD uses the acquired vehicle position information and heading angle to perform a coordinate transformation operation, transforming the first display position previously determined in the vehicle coordinate system (i.e., the initial position of the auxiliary element information relative to the vehicle) to the world coordinate system to obtain the third display position.

[0100] In this embodiment of the invention, ARHUD uses the acquired vehicle position information and heading angle as the reference point and direction for conversion, transforming the first display position (i.e., the position coordinates of the object to be prompted relative to the vehicle) previously determined in the vehicle coordinate system to the world coordinate system, which is the basis for subsequent motion prediction and time delay compensation.

[0101] Optionally, in step S442, converting the first slope value to the world coordinate system to obtain the second slope value includes the following steps:

[0102] Step S4423: Obtain the relative angle between the vehicle and the ground in the world coordinate system;

[0103] Step S4424: Based on the relative angle, the first slope value is converted to the world coordinate system to obtain the second slope value.

[0104] In this embodiment of the invention, the relative angle can be understood as the tilt angle of the vehicle relative to the horizontal ground in the world coordinate system.

[0105] Obtaining the relative angle between the vehicle and the ground in the world coordinate system can be understood as ARHUD accurately measuring the tilt angle of the vehicle relative to the horizontal ground through sensors built into the vehicle (such as gyroscopes and accelerometers), i.e., the relative angle.

[0106] The transformation of the first slope value to the world coordinate system based on the relative angle can be understood as follows: ARHUD uses the relative angle to transform the first slope value (which represents the ground tilt in the direction of vehicle travel) originally measured in the vehicle coordinate system to the world coordinate system to obtain the second slope value.

[0107] In this embodiment of the invention, the first slope value measured in the vehicle coordinate system is transformed to the world coordinate system by the relative angle. The resulting second slope value can more accurately reflect the ground tilt during the vehicle's driving process. It is also the basis for the ARHUD system to perform time delay compensation and dynamic display content adjustment, ensuring a high degree of consistency between the display of virtual information and the actual movement state of the vehicle, thereby improving driving safety and experience.

[0108] Optionally, transforming the first slope value to the world coordinate system based on the relative angle to obtain the second slope value includes the following steps:

[0109] The first slope value is transformed into the world coordinate system based on the relative angle to obtain the third slope value;

[0110] The third slope value is filtered to obtain the second slope value.

[0111] In this embodiment of the invention, the transformation of the first slope value to the world coordinate system based on the relative angle to obtain the third slope value can be understood as follows: ARHUD transforms the first slope value measured in the vehicle coordinate system according to the relative angle between the vehicle and the world coordinate system, and adjusts it to the world coordinate system to obtain the third slope value.

[0112] Filtering the third slope value to obtain the second slope value can be understood as follows: after obtaining the transformed third slope value, ARHUD further performs data filtering and processing operations. The purpose is to filter out the effective slope information that best matches the current driving environment and conditions from the third slope value, thereby obtaining the second slope value. For example, the filtering process may include removing outliers, applying filtering algorithms to smooth data fluctuations, or dynamically adjusting the slope value according to the vehicle's speed and direction. By timely updating and eliminating invalid or irrelevant information, the system ensures that it can continuously provide augmented reality displays that closely match the vehicle's current state, enhancing the driver's driving experience and safety. No limitations are specified here.

[0113] In this embodiment of the invention, ARHUD utilizes the relative angle information of the vehicle to transform the first slope value obtained in the vehicle coordinate system and adjust it to the world coordinate system, thereby obtaining a third slope value. This ensures the accurate correspondence of slope information between different coordinate systems. Subsequently, the system filters the third slope value to remove abnormal data, smooth data fluctuations, and adapt to the dynamic changes of the vehicle, ultimately obtaining a second slope value. This ensures that the virtual information can accurately reflect the real-time driving path of the vehicle, providing the driver with more intuitive and safer driving assistance information.

[0114] Optionally, determining the horizontal motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative time delay, and second slope value between the object to be prompted and the vehicle includes the following steps:

[0115] Get the first relative velocity between the object to be prompted and the vehicle in the vehicle's coordinate system;

[0116] Transform the first relative velocity into the world coordinate system to obtain the second relative velocity;

[0117] Multiply the second relative velocity by the relative time delay to obtain the predicted motion vector of the object to be prompted in the world coordinate system;

[0118] The predicted motion vector is projected onto the horizontal plane based on the second slope value to obtain the horizontal plane motion vector.

[0119] In this embodiment of the invention, the predicted motion vector can be understood as a vector representation that estimates the future position change of the object (such as a neighboring vehicle) based on the speed and time delay of the current object relative to the vehicle.

[0120] Obtaining the first relative velocity between the object to be prompted and the vehicle in the vehicle's coordinate system can be understood as follows: In the vehicle's coordinate system, ARHUD first measures and records the relative velocity between the object to be prompted (e.g., the car in front) and the vehicle, that is, the difference in velocity between the object to be prompted and the vehicle.

[0121] Transforming the first relative velocity to the world coordinate system to obtain the second relative velocity can be understood as ARHUD transforming the first relative velocity in the vehicle coordinate system to the world coordinate system to obtain the second relative velocity.

[0122] Multiplying the second relative velocity by the relative time delay to obtain the predicted motion vector of the object to be prompted in the world coordinate system can be understood as ARHUD using the second relative velocity and the relative time delay from perception to display to calculate the expected displacement of the vehicle next to it within this time delay.

[0123] Based on the second slope value, the predicted motion vector is projected onto the horizontal plane to obtain the horizontal plane motion vector. This can be understood as, taking into account the slope changes of the vehicle's travel, ARHUD projects the predicted motion vector onto the horizontal plane to generate the horizontal plane motion vector.

[0124] In this embodiment of the invention, ARHUD captures the first relative velocity between the object to be prompted (such as a neighboring vehicle) and the vehicle in the vehicle's coordinate system. By converting the first relative velocity to the world coordinate system, a second relative velocity is obtained. ARHUD multiplies the second relative velocity by the relative time delay to generate a predicted motion vector. Considering the slope changes during driving, the system further projects the predicted motion vector onto the horizontal plane based on the second slope value to generate a horizontal plane motion vector. This ensures that the content displayed by the auxiliary element accurately corresponds to the dynamics of the real world, improving the accuracy and reliability of driving assistance.

[0125] Optionally, transforming the fourth display position to the vehicle's real-time coordinate system based on the vehicle's six degrees of freedom information to obtain the second display position includes the following steps:

[0126] The vehicle's real-time coordinate system is determined based on the vehicle's six degrees of freedom information;

[0127] The fourth display position is converted to the real-time coordinate system of the vehicle to obtain the second display position.

[0128] In this embodiment of the invention, the vehicle's real-time coordinate system can be understood as a dynamic coordinate framework established based on the vehicle's current six degrees of freedom information. Exemplarily, the vehicle's real-time coordinate system can be continuously updated as the vehicle's driving state changes. Whenever the vehicle's position, direction, or attitude changes, the real-time coordinate system will also be adjusted accordingly to maintain consistency with the vehicle's actual state; this is not limited here.

[0129] Determining the vehicle's real-time coordinate system based on its six degrees of freedom information can be understood as using the six degrees of freedom information (including three-dimensional position and three-dimensional attitude changes) collected by the vehicle's sensors (such as inertial measurement units and global positioning systems) to construct a coordinate system that perfectly matches the vehicle's current driving state. This real-time coordinate system not only considers the vehicle's position (X, Y, Z) but also accurately reflects changes in its pitch, roll, and yaw angles. This ensures that regardless of the vehicle's dynamic conditions (straight driving, turning, uphill / downhill, etc.), the coordinate system can reflect its attitude and position in real time, providing a dynamic and accurate reference framework for subsequent coordinate transformations and information display.

[0130] Transforming the fourth display position to the vehicle's real-time coordinate system to obtain the second display position can be understood as transforming the fourth display position to a real-time coordinate system synchronized with the vehicle's status in order to obtain display position information that is more compatible with the vehicle's current driving environment, i.e., the second display position.

[0131] In this embodiment of the invention, by integrating the vehicle's six degrees of freedom information, including the vehicle's three-dimensional position and three-dimensional attitude (pitch, roll, yaw), ARHUD dynamically constructs a real-time coordinate system for the vehicle. The pre-processed fourth display position is then transformed into the real-time coordinate system of the vehicle, thereby obtaining the second display position. This ensures that the position information of the virtual content is completely adjusted according to the real-time state of the vehicle, thus presenting the most accurate and fitting augmented reality information in the driver's current field of vision, greatly improving the real-time performance and accuracy of driving assistance.

[0132] Figure 5 This is a schematic diagram illustrating the adjustment of displayed content based on slope information according to an embodiment of the present invention, as shown below. Figure 5As shown, firstly, ARHUD converts the perceived objects in the vehicle coordinate system into objects in the world coordinate system as observation data. This involves acquiring the perceived objects in the vehicle coordinate system and then using the vehicle's position and heading angle in the world coordinate system to obtain the perceived objects in the world coordinate system. Secondly, ARHUD converts the corresponding slope values ​​in the vehicle coordinate system into slope values ​​in the world coordinate system. This is done by converting the slope values ​​identified in the vehicle coordinate system to the world coordinate system based on the relative angles and then filtering them to determine the target slope value. Next, ARHUD performs three-dimensional dynamic delay compensation for dynamic obstacles (following vehicles) in the world coordinate system based on the slope information. This involves multiplying the relative speed and relative time delay between the vehicle and the following vehicle in the world coordinate system to obtain a predicted motion vector. Then, based on the slope, the predicted motion vector is projected onto the horizontal plane to obtain the horizontal plane motion vector. Finally, ARHUD transforms the position information of the adjacent vehicle in the world coordinate system to the real-time coordinate system of the vehicle based on the six degrees of freedom information of the vehicle itself, thus obtaining the coordinate information of the adjacent vehicle's position in the real-time coordinate system of the vehicle. As a result, it can present the most accurate and fitting augmented reality information in the driver's current field of vision, greatly improving the real-time performance and accuracy of driving assistance, and providing the driver with a more intuitive and safer driving assistance experience.

[0133] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0134] According to an embodiment of the present invention, an embodiment of an augmented reality head-up display device for vehicle auxiliary element information is provided. It should be noted that the device can be used to execute the above-described augmented reality head-up display method for vehicle auxiliary element information.

[0135] Figure 6 This is a schematic diagram of an augmented reality head-up display device for vehicle auxiliary element information according to an embodiment of the present invention, such as... Figure 6As shown, the augmented reality head-up display device 600 for vehicle auxiliary element information includes: an acquisition module 601, used to acquire environmental perception data of the vehicle and the corresponding slope value of the vehicle in response to the vehicle being in augmented reality auxiliary element prompting mode, wherein the change in slope value is used to reflect the shaking state of the vehicle; a determination module 602, used to determine auxiliary element information based on environmental perception data, wherein the auxiliary element information includes a prompting image of the object to be prompted; a time delay compensation module 603, used to perform time delay compensation on the first display position of the auxiliary element information based on the slope value to obtain a second display position; and a display module 604, used to display the auxiliary element information at the second display position within the augmented reality head-up display field of view of the vehicle.

[0136] Furthermore, the latency compensation module 603 is also used to obtain the first display position of the object to be prompted in the vehicle coordinate system, and the first slope value of the vehicle in the vehicle coordinate system; to convert the first display position to the world coordinate system to obtain the third display position, and to convert the first slope value to the world coordinate system to obtain the second slope value; to determine the horizontal motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative latency, and second slope value between the object to be prompted and the vehicle, wherein the horizontal motion vector is used to represent the motion direction of the object to be prompted and the distance offset of the object to be prompted, and the relative latency includes the dynamic latency of the prompt image of the object to be prompted during the acquisition, transmission, and calculation process, as well as the fixed latency during the rendering and display process; to determine the fourth display position of the object to be prompted in the world coordinate system based on the third display position and the horizontal motion vector; and to convert the fourth display position to the vehicle real-time coordinate system based on the six degrees of freedom information of the vehicle to obtain the second display position.

[0137] Furthermore, the delay compensation module 603 is also used to obtain the vehicle's position information and heading angle in the world coordinate system; based on the vehicle's position information and heading angle, the first display position is transformed to the world coordinate system to obtain the third display position.

[0138] Furthermore, the time delay compensation module 603 is also used to obtain the relative angle between the vehicle and the ground in the world coordinate system; based on the relative angle, the first slope value is converted to the world coordinate system to obtain the second slope value.

[0139] Furthermore, the time delay compensation module 603 is also used to convert the first slope value to the world coordinate system based on the relative angle to obtain the third slope value; and to filter the third slope value to obtain the second slope value.

[0140] Furthermore, the time delay compensation module 603 is also used to obtain the first relative velocity between the object to be prompted and the vehicle in the vehicle coordinate system; convert the first relative velocity to the world coordinate system to obtain the second relative velocity; multiply the second relative velocity by the relative time delay to obtain the predicted motion vector of the object to be prompted in the world coordinate system; and project the predicted motion vector onto the horizontal plane based on the second slope value to obtain the horizontal plane motion vector.

[0141] Furthermore, the time delay compensation module 603 is also used to determine the real-time coordinate system of the vehicle based on the vehicle's six degrees of freedom information; and to transform the fourth display position to the real-time coordinate system of the vehicle to obtain the second display position.

[0142] Furthermore, the display module 604 is also used to display the object to be prompted, including static objects or dynamic objects. Dynamic objects include at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian, or an animal within the field of view of the augmented reality head-up display of the vehicle. Static objects include at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian, or an animal within the field of view of the augmented reality head-up display of the vehicle.

[0143] According to another aspect of the present invention, a vehicle is also provided, which is used to perform the augmented reality head-up display method for vehicle auxiliary element information in various embodiments of the present invention.

[0144] According to another aspect of the present invention, an electronic device is also provided, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform an augmented reality head-up display method for vehicle auxiliary element information according to various embodiments of the present invention.

[0145] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored executable program, wherein, when the executable program is executed, it controls the device where the computer-readable storage medium is located to perform the augmented reality head-up display method for vehicle auxiliary element information in various embodiments of the present invention.

[0146] According to another aspect of the present invention, a computer program product is also provided, including a computer program that, when executed by a processor, implements the augmented reality head-up display method for vehicle auxiliary element information in various embodiments of the present invention.

[0147] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0148] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0149] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0150] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0151] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0152] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An augmented reality head-up display method for vehicle auxiliary element information, characterized in that, include: In response to the vehicle being in augmented reality assistive element prompting mode, the environmental perception data of the vehicle and the slope value corresponding to the vehicle are acquired, wherein the change in the slope value is used to reflect the shaking state of the vehicle; Based on the environmental perception data, auxiliary element information is determined, wherein the auxiliary element information includes a prompting image of the object to be prompted; Based on the slope value, the first display position of the auxiliary element information is compensated for time delay to obtain the second display position; The auxiliary element information is displayed at the second display position within the field of view of the augmented reality head-up display of the vehicle; The step of performing time delay compensation on the first display position of the auxiliary element information based on the slope value to obtain the second display position includes: obtaining the first display position of the object to be prompted in the vehicle coordinate system, and the first slope value corresponding to the vehicle in the vehicle coordinate system; converting the first display position to the world coordinate system to obtain the third display position, and converting the first slope value to the world coordinate system to obtain the second slope value; determining the horizontal plane motion vector of the object to be prompted in the world coordinate system based on the relative speed and relative time delay between the object to be prompted and the vehicle, and the second slope value, wherein the horizontal plane motion vector is used to represent the motion direction of the object to be prompted and the distance offset of the object to be prompted, and the relative time delay includes the dynamic time delay of the prompt image of the object to be prompted during the acquisition, transmission and calculation process, and the fixed time delay during the rendering and display process; determining the fourth display position of the object to be prompted in the world coordinate system based on the third display position and the horizontal plane motion vector; and converting the fourth display position to the vehicle real-time coordinate system based on the six degrees of freedom information of the vehicle to obtain the second display position.

2. The method according to claim 1, characterized in that, The step of converting the first display position to the world coordinate system to obtain the third display position includes: Obtain the vehicle's position information and heading angle in the world coordinate system; Based on the vehicle's position information and the heading angle, the first display position is transformed into the world coordinate system to obtain the third display position.

3. The method according to claim 1, characterized in that, The step of converting the first slope value to the world coordinate system to obtain the second slope value includes: Obtain the relative angle between the vehicle and the ground in the world coordinate system; Based on the relative angle, the first slope value is converted to the world coordinate system to obtain the second slope value.

4. The method according to claim 3, characterized in that, The step of converting the first slope value to the world coordinate system based on the relative angle to obtain the second slope value includes: Based on the relative angle, the first slope value is converted to the world coordinate system to obtain the third slope value; The third slope value is filtered to obtain the second slope value.

5. The method according to claim 1, characterized in that, The step of determining the horizontal motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative time delay, and the second slope value between the object to be prompted and the vehicle includes: Obtain the first relative velocity between the object to be prompted and the vehicle in the vehicle coordinate system; The first relative velocity is transformed into the world coordinate system to obtain the second relative velocity; Multiplying the second relative velocity by the relative time delay yields the predicted motion vector of the object to be prompted in the world coordinate system. The predicted motion vector is projected onto the horizontal plane based on the second slope value to obtain the horizontal plane motion vector.

6. The method according to claim 1, characterized in that, The process of transforming the fourth display position to the vehicle's real-time coordinate system based on the vehicle's six degrees of freedom information to obtain the second display position includes: The real-time coordinate system of the vehicle is determined based on the six degrees of freedom information of the vehicle. The fourth display position is converted to the real-time coordinate system of the vehicle to obtain the second display position.

7. The method according to any one of claims 1-6, characterized in that, The object to be prompted includes static objects or dynamic objects. The dynamic objects include at least one of the following: a moving motor vehicle, a moving non-motor vehicle, a moving pedestrian, or an animal within the field of view of the augmented reality head-up display of the vehicle. The static objects include at least one of the following: a non-moving motor vehicle, a non-moving non-motor vehicle, an obstacle, a non-moving pedestrian, or an animal within the field of view of the augmented reality head-up display of the vehicle.

8. An augmented reality head-up display device for vehicle auxiliary element information, characterized in that, include: The acquisition module is used to acquire the environmental perception data of the vehicle and the slope value corresponding to the vehicle in response to the vehicle being in augmented reality assistive element prompting mode, wherein the change in the slope value is used to reflect the shaking state of the vehicle. The determination module is used to determine auxiliary element information based on the environmental perception data, wherein the auxiliary element information includes a prompting image of the object to be prompted; The time delay compensation module is used to perform time delay compensation on the first display position of the auxiliary element information based on the slope value to obtain the second display position; The display module is used to display the auxiliary element information at the second display position within the field of view of the augmented reality head-up display of the vehicle; The latency compensation module is further configured to: acquire the first display position of the object to be prompted in the vehicle coordinate system, and the first slope value corresponding to the vehicle in the vehicle coordinate system; convert the first display position to the world coordinate system to obtain a third display position, and convert the first slope value to the world coordinate system to obtain a second slope value; determine the horizontal motion vector of the object to be prompted in the world coordinate system based on the relative speed, relative latency, and second slope value of the object to be prompted and the vehicle, wherein the horizontal motion vector represents the motion direction and distance offset of the object to be prompted, and the relative latency includes the dynamic latency of the prompt image of the object to be prompted during acquisition, transmission, and calculation, and the fixed latency during rendering and display; determine the fourth display position of the object to be prompted in the world coordinate system based on the third display position and the horizontal motion vector; and convert the fourth display position to the vehicle real-time coordinate system based on the six degrees of freedom information of the vehicle to obtain the second display position.

9. A vehicle, characterized in that, The vehicle is used to perform the augmented reality head-up display method for vehicle auxiliary element information as described in any one of claims 1 to 8.

10. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the augmented reality head-up display method for vehicle auxiliary element information as described in any one of claims 1 to 8.

Images