AR navigation projection method and system

Through the AR image generation algorithm and navigation projection system, navigation information is dynamically adjusted and projected to the front windshield of the vehicle, solving the problem of unintuitive navigation instructions and improving navigation effect and driving safety.

CN119085700APending Publication Date: 2024-12-06WUHAN CARBIT INFORMATION CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202411154314.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-22
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The existing navigation methods have not intuitive navigation instructions, and the navigation effect is relatively average, making it difficult to meet the current consumers' safe driving needs.

Method used

Using the AR image generation algorithm, dynamically adjusts the display content and form of navigation information according to the actual driving speed, driving attitude and road conditions of the vehicle, and superimposes the navigation information on the real-time road condition image. The navigation image is projected to the vehicle through the AR navigation projection system. A predetermined area of ​​the front windshield.

Benefits of technology

It realizes intuitive display of navigation information, improves navigation display effect and real-time performance, and improves the driver's driving experience and vehicle driving safety.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119085700A_ABST
    Figure CN119085700A_ABST
Patent Text Reader

Abstract

The invention provides an AR (Augmented Reality) navigation projection method and system. The method comprises the following steps: receiving navigation information sent by a mobile terminal or a vehicle-mounted navigation system; based on the navigation information, generating a navigation image through an AR image generation algorithm; wherein the AR image generation algorithm is used for dynamically adjusting the display content and form of the navigation information according to the actual driving speed, driving posture and road condition of the vehicle, and superposing the navigation information into the real-time road condition image to obtain a navigation image; and projecting the navigation image to a predetermined area of a front windshield of the vehicle. According to the scheme, the navigation information can be visually displayed, the navigation information display effect is effectively improved, and the user driving experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of vehicle navigation, and in particular relates to an AR navigation projection method and system. Background Art

[0002] With the development of intelligent automobile technology, drivers have higher and higher requirements for vehicle navigation. Traditional in-vehicle navigation systems mostly rely on in-vehicle display screens, which have safety risks such as distracted vision and inconvenient operation, and are difficult to meet the current consumers' safe driving needs. At present, the more mature navigation projection method is to use head-up display (HUD), but this method of navigation instructions is not intuitive and the navigation effect is relatively general. Summary of the invention

[0003] In view of this, an embodiment of the present invention provides an AR navigation projection method and system, which are used to solve the problem that the navigation instructions of the existing navigation method are not intuitive and the navigation effect is relatively general.

[0004] In a first aspect of an embodiment of the present invention, an AR navigation projection method is provided, including:

[0005] Receiving navigation information sent by a mobile terminal or a vehicle navigation system;

[0006] Based on the navigation information, generating a navigation image by using an AR image generation algorithm;

[0007] The AR image generation algorithm is used to dynamically adjust the display content and form of the navigation information according to the actual driving speed, driving posture and road conditions of the vehicle, and superimpose the navigation information on the real-time road condition image to obtain the navigation image;

[0008] Projects the navigation image onto a predetermined area of ​​the vehicle's front windshield.

[0009] In a second aspect of an embodiment of the present invention, an AR navigation projection system is provided, including:

[0010] A data receiving module, used to receive navigation information sent by a mobile terminal or a vehicle navigation system;

[0011] An image generation module, used to generate a navigation image through an AR image generation algorithm based on the navigation information;

[0012] The AR image generation algorithm is used to dynamically adjust the display content and form of the navigation information according to the actual driving speed, driving posture and road conditions of the vehicle, and superimpose the navigation information on the real-time road condition image to obtain the navigation image;

[0013] The navigation projection module is used to project the navigation image onto a predetermined area of ​​the vehicle's front windshield.

[0014] In a third aspect of an embodiment of the present invention, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method described in the first aspect of the embodiment of the present invention when executing the computer program.

[0015] In a fourth aspect of an embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method provided in the first aspect of the embodiment of the present invention are implemented.

[0016] In the embodiment of the present invention, based on the AR image generation algorithm, the display content and form of the navigation information are dynamically adjusted, and the navigation information is superimposed on the real-time road condition image to obtain a navigation image, and the navigation image is projected and displayed, thereby realizing AR projection of the navigation information. It can not only intuitively display the navigation information and improve the navigation display effect, but also ensure the real-time and accuracy of the navigation, thereby improving the driver's driving experience and improving the safety of vehicle driving to a certain extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0018] Figure 1 A schematic diagram of a flow chart of an AR navigation projection method provided by an embodiment of the present invention;

[0019] Figure 2 A schematic diagram of the structure of an AR navigation projection system provided by an embodiment of the present invention;

[0020] Figure 3 A schematic structural diagram of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0021] In order to make the purpose, features and advantages of the present invention more obvious and easy to understand, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the embodiments described below are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] It should be understood that the term "including" and other similar expressions in the specification or claims of the present invention and the above-mentioned drawings are intended to cover non-exclusive inclusions, such as a process, method, system, or device including a series of steps or units is not limited to the listed steps or units. In addition, "first" and "second" are used to distinguish different objects, not to describe a specific order.

[0023] See also Figure 1 , a flowchart of an AR navigation projection method provided by an embodiment of the present invention includes:

[0024] S101, receiving navigation information sent by a mobile terminal or a vehicle navigation system;

[0025] The mobile terminal is generally a smart terminal device such as a mobile phone, and the vehicle navigation system is a navigation system owned by the vehicle itself or a third-party navigation APP installed on the vehicle. The mobile terminal can be connected to the vehicle terminal via wireless (such as Bluetooth, WiFi, etc.) or wired (such as USB, etc.) to send the navigation information on the mobile terminal to the vehicle terminal.

[0026] The navigation information includes at least route indication information, such as left turn, right turn, straight ahead, U-turn, lane change, etc., and may also include road condition forecasts ahead, speed limit prompts, etc.

[0027] S102, generating a navigation image through an AR image generation algorithm based on the navigation information;

[0028] The AR image generation algorithm is used to dynamically adjust the display content and form of the navigation information according to the actual driving speed, driving posture and road conditions of the vehicle, and superimpose the navigation information on the real-time road condition image to obtain the navigation image;

[0029] The vehicle's longitude, latitude, speed, direction angle and other basic information can be obtained through the on-board GPS receiver. Based on the vehicle's wheel speed meter, accelerometer, gyroscope and other sensors, the vehicle's speed, acceleration, attitude and other information can be obtained through sensor fusion algorithms (such as Kalman filtering, etc.), and the vehicle speed, attitude and other information in the GPS can be calibrated.

[0030] Road condition information can be obtained by collecting real-time road images in front of the vehicle through the on-board camera and identifying traffic element information in the image through deep learning models. At the same time, navigation information can also provide forecasts of road conditions ahead of the vehicle, such as intersections and sharp turns ahead. Road condition information ahead can be obtained based on the on-board camera and navigation information.

[0031] Among them, the original image captured by the camera is preprocessed by denoising, defogging, contrast enhancement and other operations to improve the image quality.

[0032] Time synchronization is performed on GPS data, sensor data, and on-board camera image data to ensure the accuracy of vehicle speed and posture calculations, and to ensure that navigation information can be accurately superimposed on road condition images.

[0033] The display content and form of navigation information are dynamically adjusted according to the vehicle speed, posture and road condition information. For example, when the vehicle approaches an intersection and needs to go straight, a dotted arrow can be used to indicate that the vehicle is slowing down. When the vehicle needs to turn right, a right turn icon can be displayed. The closer the vehicle is to the intersection and the faster the vehicle speed, the more obvious the sign will be. At the same time, when the vehicle speed exceeds a certain value, a specific right turn indicator can be used to guide the vehicle to slow down and turn right. When the vehicle posture changes significantly, the display form of the navigation information will also change accordingly.

[0034] Specifically, based on the vehicle navigation information, vehicle positioning and real-time image recognition information, it is determined whether the vehicle is close to a predetermined key area; if it is determined that the vehicle is close to the predetermined key area, the navigation information is displayed in the form of a predetermined indicator icon, and the size and brightness of the predetermined indicator icon are adjusted according to the distance between the vehicle and the predetermined key area, the vehicle speed and the vehicle posture; the predetermined key area includes at least intersections, crosswalks, and construction areas.

[0035] The preset transparency of navigation information display is obtained, and the navigation information is adjusted to a corresponding transparency and then superimposed on the real-time traffic image.

[0036] Optionally, a deep learning model is used to identify road signs and traffic signals in real-time traffic images, and corresponding road signs and traffic signals are marked in the navigation image.

[0037] A deep learning model (such as convolutional neural network CNN) is trained by using a large amount of annotated road signs and traffic signal image data. The images collected by the camera are pre-processed and input into the trained model for real-time recognition. The recognition results (such as speed limit signs, no left turn signs, traffic light status, etc.) are marked on the navigation image in a striking way to provide drivers with timely road condition information. User feedback on the recognition results is collected and used to continuously optimize and update the recognition model.

[0038] S103: Projecting the navigation image onto a predetermined area of ​​the front windshield of the vehicle.

[0039] The projection of the navigation image is achieved through an AR navigation image projection device. The navigation image is restored by encoding the light source projection and then through light interference and illumination.

[0040] The light source generally uses low-power, high-brightness and excellent color reproduction LED as the projection light source. The LED light source not only has low energy consumption and long life, but also can produce uniform and bright light, providing clear and bright color performance for the projected image. In order to ensure that the LED light source works stably for a long time, an effective heat dissipation system needs to be designed, such as using heat sinks, fans or heat pipes and other heat dissipation technologies to avoid performance degradation or shortened life caused by overheating of the light source. The lens group uses low-dispersion, high-transmittance glass or optical plastic as the lens material to reduce the loss and dispersion of light during the transmission process.

[0041] The lens configuration consists of multiple lenses, including convex lenses, concave lenses, and possibly aspheric lenses. The curvature, thickness, and distance between each lens are precisely calculated to ensure that the light can form a clear, distortion-free image after passing through the lens group. The light path can be simulated and optimized using optical design software (such as Zemax, Code V, etc.) to ensure that the light is emitted from the projection light source, refracted and reflected by the lens group, and finally projected onto the windshield according to the predetermined path, and forms the desired image size, shape, and position.

[0042] The reflector is made of a coating material with high reflectivity and low absorption, such as aluminum or silver film, to ensure that the light is minimally lost during the reflection process. The shape and angle of the reflector can be designed according to the requirements of the optical system to control the reflection direction and focusing effect of the light. To ensure that the reflector can be stably installed in the system, fine-tuning is performed through a precise adjustment mechanism to optimize the reflection effect and image quality.

[0043] Preferably, based on an angle sensor and a distance sensor installed below the front windshield of the vehicle or at a set position inside the vehicle, the inclination angle of the windshield and the driver's line of sight height are detected; according to the inclination angle of the windshield and the driver's line of sight height, the projection angle and height of the AR navigation image projection device are adjusted by motor drive.

[0044] An automatic adjustment mechanism is used to adjust the projection angle and height of the projection device, which receives sensor data and converts it into mechanical movement. The adjustment mechanism may include components such as a motor, a gear transmission system, and a slide rail. The accuracy and reliability of the projection angle adjustment are ensured based on the adjustment mechanism. At the same time, a closed-loop control system is introduced to monitor and adjust the adjustment effect in real time through a sensor feedback mechanism to improve the adjustment accuracy and response speed.

[0045] Preferably, a light signal collected by a light sensor in the vehicle is received, and the brightness and contrast of a projection light source of the AR navigation image projection device are controlled according to the light signal.

[0046] Considering the variation range of light intensity in different weather and time periods as well as the driver's visual comfort requirements, corresponding adjustment strategies are formulated to automatically and adaptively adjust the brightness and contrast of the projection light source.

[0047] In some embodiments, a learning mechanism is introduced to optimize and improve the brightness and contrast adjustment algorithm of the projection light source by continuously collecting and analyzing the usage data and feedback from drivers.

[0048] In other embodiments, a projection light source control system is designed, which can receive the output signal of the adjustment algorithm and control the brightness and contrast adjustment circuit of the projection light source. The control system has high-precision PWM (pulse width modulation) or DAC (digital-to-analog converter) output capability to achieve fine adjustment of the brightness and contrast of the projection light source.

[0049] In this embodiment, the navigation information is superimposed on the real-time traffic image through the AR image generation algorithm, and the generated navigation image is AR projected, which not only can make the navigation information intuitive and improve the navigation display effect, but also can ensure that the navigation information is displayed in real time and accurately, thereby improving the driver's experience.

[0050] In one embodiment, a voice command or a gesture command input by a user is acquired through voice recognition or gesture recognition, a navigation route is switched according to the voice command or the gesture command, and corresponding navigation information is updated.

[0051] The user can control the switching of navigation routes and the navigation display content through simple voice commands or gesture operations. For example, if the left turn ahead is switched to straight ahead, the corresponding navigation image will also change. Through voice commands and gesture commands, the driver can directly switch the navigation route without manual operation.

[0052] It should be understood that the serial numbers of the steps in the above embodiments do not imply a sequence of execution, and the execution sequence of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0053] Figure 2 A schematic diagram of the structure of an AR navigation projection system provided by an embodiment of the present invention, the system includes:

[0054] The data receiving module 210 is used to receive navigation information sent by a mobile terminal or a vehicle navigation system;

[0055] An image generation module 220, configured to generate a navigation image based on the navigation information by using an AR image generation algorithm;

[0056] The AR image generation algorithm is used to dynamically adjust the display content and form of the navigation information according to the actual driving speed, driving posture and road conditions of the vehicle, and superimpose the navigation information on the real-time road condition image to obtain the navigation image;

[0057] Specifically, based on vehicle navigation information, vehicle positioning and real-time image recognition information, it is determined whether the vehicle is close to a predetermined key area;

[0058] If it is determined that the vehicle is close to a predetermined key area, the navigation information is displayed in the form of a predetermined indicator icon, and the size and brightness of the predetermined indicator icon are adjusted according to the distance between the vehicle and the predetermined key area, the vehicle speed and the vehicle posture;

[0059] The predetermined key area includes at least intersections and crosswalks.

[0060] The preset transparency of navigation information display is obtained, and the navigation information is adjusted to a corresponding transparency and then superimposed on the real-time traffic image.

[0061] Optionally, the image generation module 220 includes:

[0062] The recognition and annotation unit is used to identify road signs and traffic signals in real-time traffic images through a deep learning model, and to annotate the corresponding road signs and traffic signals in the navigation image.

[0063] The navigation projection module 230 is used to project the navigation image onto a predetermined area of ​​the front windshield of the vehicle.

[0064] Preferably, the navigation projection module 230 includes:

[0065] A detection unit, for detecting the tilt angle of the windshield and the sight height of the driver based on an angle sensor and a distance sensor installed below the front windshield of the vehicle or at a set position inside the vehicle;

[0066] The adjustment unit is used to adjust the projection angle and height of the AR navigation image projection device through motor drive according to the inclination angle of the windshield and the driver's line of sight height.

[0067] Preferably, the navigation projection module 230 includes:

[0068] The optical control unit is used to receive the light signal collected by the light sensor in the vehicle and control the brightness and contrast of the projection light source of the AR navigation image projection device according to the light signal.

[0069] In one embodiment, the image generation module 220 further includes:

[0070] The route switching module is used to obtain the voice command or gesture command input by the user through voice recognition or gesture recognition, switch the navigation route according to the voice command or gesture command, and update the corresponding navigation information.

[0071] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described systems and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0072] Figure 3 1 is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. The electronic device is used for AR navigation. Figure 3 As shown, the electronic device 3 of this embodiment includes: a memory 310, a processor 320 and a system bus 330, wherein the memory 310 includes an executable program 3101 stored thereon, and those skilled in the art can understand that Figure 3 The electronic device structure shown in the figure does not constitute a limitation of the electronic device, and may include more or less components than shown in the figure, or combine certain components, or arrange the components differently.

[0073] Combine the following Figure 3 A detailed introduction to the various components of electronic equipment:

[0074] The memory 310 can be used to store software programs and modules, and the processor 320 executes various functional applications and data processing of the electronic device by running the software programs and modules stored in the memory 310. The memory 310 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as cache data), etc. In addition, the memory 310 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices.

[0075] The memory 310 includes an executable program 3101 of a network request method, and the executable program 3101 can be divided into one or more modules / units, which are stored in the memory 310 and executed by the processor 320 to implement AR navigation projection, etc. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program 3101 in the electronic device 3. For example, the computer program 3101 can be divided into functional modules such as a data receiving module, an image generation module, and a navigation projection module.

[0076] The processor 320 is the control center of the electronic device. It uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 310, and calling data stored in the memory 310, it performs various functions of the electronic device and processes data, thereby monitoring the overall status of the electronic device. Optionally, the processor 320 may include one or more processing units; preferably, the processor 320 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, application programs, etc., and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 320.

[0077] The system bus 330 is used to connect the various functional components inside the computer, and can transmit data information, address information, and control information. Its types can be, for example, PCI bus, ISA bus, CAN bus, etc. The instructions of the processor 320 are transmitted to the memory 310 through the bus, and the memory 310 feeds back data to the processor 320. The system bus 330 is responsible for the data and instruction exchange between the processor 320 and the memory 310. Of course, the system bus 330 can also be connected to other devices, such as network interfaces, display devices, etc.

[0078] In the embodiment of the present invention, the executable program executed by the processor 320 included in the electronic device includes:

[0079] Receiving navigation information sent by a mobile terminal or a vehicle navigation system;

[0080] Based on the navigation information, generating a navigation image by using an AR image generation algorithm;

[0081] The AR image generation algorithm is used to dynamically adjust the display content and form of the navigation information according to the actual driving speed, driving posture and road conditions of the vehicle, and superimpose the navigation information on the real-time road condition image to obtain the navigation image;

[0082] Projects the navigation image onto a predetermined area of ​​the vehicle's front windshield.

[0083] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0084] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0085] As described above, the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. An AR navigation projection method, It is characterized in that include: Receiving navigation information sent by a mobile terminal or a vehicle navigation system; Based on the navigation information, generating a navigation image by using an AR image generation algorithm; The AR image generation algorithm is used to dynamically adjust the display content and form of the navigation information according to the actual driving speed, driving posture and road conditions of the vehicle, and superimpose the navigation information on the real-time road condition image to obtain the navigation image; Projects the navigation image onto a predetermined area of ​​the vehicle's front windshield.

2. The method according to claim 1, It is characterized in that The dynamically adjusting the display content and form of the navigation information according to the actual driving speed and road conditions of the vehicle includes: Based on vehicle navigation information, vehicle positioning and real-time image recognition information, determine whether the vehicle is close to a predetermined key area; If it is determined that the vehicle is close to a predetermined key area, the navigation information is displayed in the form of a predetermined indicator icon, and the size and brightness of the predetermined indicator icon are adjusted according to the distance between the vehicle and the predetermined key area, the vehicle speed and the vehicle posture; The predetermined key area includes at least intersections and crosswalks.

3. The method according to claim 1, It is characterized in that The step of superimposing the navigation information onto the real-time traffic image to obtain the navigation image comprises: Get the preset navigation information display transparency, adjust the navigation information to the corresponding transparency, and then overlay it on the real-time traffic image.

4. The method according to claim 1, It is characterized in that The step of superimposing the navigation information onto the real-time traffic image to obtain the navigation image further comprises: The deep learning model is used to identify road signs and traffic signals in real-time traffic images, and the corresponding road signs and traffic signals are marked in the navigation images.

5. The method according to claim 1, It is characterized in that The generating a navigation image by using an AR image generating algorithm based on the navigation information further includes: The voice command or gesture command input by the user is obtained through voice recognition or gesture recognition, the navigation route is switched according to the voice command or gesture command, and the corresponding navigation information is updated.

6. The method according to claim 1, It is characterized in that The step of projecting the navigation image onto a predetermined area of ​​the vehicle's front windshield comprises: Based on an angle sensor and a distance sensor installed below the front windshield of the vehicle or at a set position inside the vehicle, the tilt angle of the windshield and the driver's sight height are detected; According to the inclination angle of the windshield and the driver's line of sight height, the projection angle and height of the AR navigation image projection device are adjusted by motor drive.

7. The method according to claim 1, It is characterized in that The projecting of the navigation image onto a predetermined area of ​​the front windshield of the vehicle further comprises: Receive the light signal collected by the light sensor in the car, and control the brightness and contrast of the projection light source of the AR navigation image projection device according to the light signal.

8. An AR navigation projection system, It is characterized in that include: A data receiving module, used to receive navigation information sent by a mobile terminal or a vehicle navigation system; An image generation module, used to generate a navigation image through an AR image generation algorithm based on the navigation information; The AR image generation algorithm is used to dynamically adjust the display content and form of the navigation information according to the actual driving speed, driving posture and road conditions of the vehicle, and superimpose the navigation information on the real-time road condition image to obtain the navigation image; The navigation projection module is used to project the navigation image onto a predetermined area of ​​the vehicle's front windshield.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, It is characterized in that When the processor executes the computer program, the steps of the AR navigation projection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program. It is characterized in that When the computer program is executed, the steps of an AR navigation projection method as described in any one of claims 1 to 7 are implemented.

Citation Information

Cited By

  • Immersive navigation method and system and electronic equipment

    CN120927023A