Augmented reality navigation image rendering method and device, electronic device, and storage medium

Abstract

The application discloses an AR navigation image rendering method and device, electronic equipment and a storage medium, and is used for rendering a real scene image in a video frame. The method comprises the following steps: acquiring a current frame in the video frame; rendering the current frame, and recording a starting time and an ending time of the rendering; acquiring first driving state information and second driving state information of the vehicle at the starting time and the ending time respectively; calculating the latitude and longitude of the vehicle when the next frame is acquired based on the starting time, the first driving state information, the ending time, the second driving state information and the latitude and longitude of the vehicle when the current frame is acquired; and rendering the next frame based on the calculated latitude and longitude of the vehicle when the next frame is acquired. The application reduces the algorithm complexity and the power consumption, reduces the calculation error and improves the accuracy. The latitude and longitude are used as the reference to participate in the modeling calculation, so that the model can be real-time fitted with the actual situation.

Description

Technical Field

[0001] This invention relates to the field of vehicle AR navigation technology, specifically to an AR navigation image rendering method, apparatus, electronic device, and storage medium. Background Technology

[0002] Augmented Reality (AR) technology is a technology that uses camera images to calculate the objects and angles in the images and then models them. After modeling, virtual images, videos, or 3D models are added and overlaid to provide users with a visual experience that combines the virtual and real worlds.

[0003] For vehicles, compared to traditional navigation, AR navigation uses cameras to capture the real-world scene of the road ahead, then combines this with the vehicle's current location, map information, and scene recognition data to generate virtual guidance information on the visible real road. The successful integration of virtual and real elements in in-vehicle AR navigation relies on vehicle positioning technology, image detection and recognition technology, and AR fusion technology.

[0004] However, in the existing technology, when a vehicle equipped with an AR navigation system is in motion, the positioning matching update frequency is low, and the positioning update frequency is limited by the vehicle system. Currently, the most commonly used positioning update frequency is 1Hz, which cannot meet the 30Hz rendering frequency required by AR navigation. As a result, the rendered object cannot accurately match the reality at all times. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to overcome the defect in the prior art that the rendered object of the vehicle equipped with the AR navigation system cannot accurately match the reality at all times during the driving process, and to provide an AR navigation image rendering method, device, electronic device and storage medium.

[0006] The present invention solves the above-mentioned technical problems through the following technical solution:

[0007] This invention provides an AR navigation image rendering method, which is used to render real-world images in video frames. The method includes: obtaining the current frame in the video frame;

[0008] Render the current frame and record the start and end times of the rendering.

[0009] The first driving status information and the second driving status information of the vehicle are obtained at the start time and the end time, respectively.

[0010] The latitude and longitude of the vehicle at the time of acquiring the next frame are calculated based on the start time, the first driving state information, the end time, the second driving state information, and the latitude and longitude of the vehicle at the time of acquiring the current frame.

[0011] The next frame is rendered based on the calculated latitude and longitude of the vehicle when the next frame is acquired.

[0012] The present invention also provides an AR navigation image rendering device, the device being used to render real-world images in video frames, the device comprising:

[0013] The video frame acquisition module is used to acquire the current frame in a video frame.

[0014] The rendering module is used to render the current frame and record the start and end times of the rendering.

[0015] The data acquisition module is used to acquire the first driving state information and the second driving state information of the vehicle at the start time and the end time, respectively.

[0016] The processing module is used to calculate the latitude and longitude of the vehicle when the current frame is acquired based on the start time, the first driving state information, the end time, the second driving state information, and the latitude and longitude of the vehicle when the next frame is acquired.

[0017] The rendering module is also used to render the next frame based on the calculated latitude and longitude of the vehicle when the next frame is acquired.

[0018] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the aforementioned method.

[0019] The present invention also provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to perform the aforementioned method.

[0020] The positive and progressive effects of this invention are as follows:

[0021] In-vehicle AR navigation ensures that rendered objects accurately match reality in real time while the vehicle is in motion: it uses gyroscopes to stably acquire vehicle driving status information (including speed, direction, acceleration, etc.) to calculate latitude and longitude in real time, reducing algorithm complexity and power consumption. It also uses latitude and longitude matched by the map to calibrate and reduce calculation errors, thereby improving accuracy. Using this latitude and longitude as a benchmark in the modeling calculation, it can ensure that the model matches reality in real time.

[0022] With the help of technologies such as intelligent positioning, map navigation, and AR rendering, more intuitive 3D real-world guidance can be presented on real-world road footage, helping users with poor sense of direction to solve problems such as difficulty in finding direction when starting and not knowing when to turn. Attached Figure Description

[0023] Figure 1 This is a flowchart of the AR navigation image rendering method according to Embodiment 1 of the present invention.

[0024] Figure 2 This is a flowchart of step S104 in Embodiment 1 of the present invention.

[0025] Figure 3 This is a schematic diagram of the AR navigation image rendering device according to Embodiment 2 of the present invention.

[0026] Figure 4 This is a schematic diagram of the processing module in Embodiment 2 of the present invention.

[0027] Figure 5 This is a schematic diagram of the electronic device according to Embodiment 3 of the present invention. Detailed Implementation

[0028] The present invention will be further illustrated by way of embodiments below, but the present invention is not limited to the scope of the embodiments described herein.

[0029] Example 1

[0030] like Figure 1 As shown, this embodiment discloses an AR navigation image rendering method, which is used to render real-world images in video frames. The method includes:

[0031] Step S101: Obtain the current frame in the video frame;

[0032] Specifically, the current frame is the frame that is being rendered.

[0033] Step S102: Render the current frame and record the start and end times of the rendering.

[0034] Specifically, the rendered objects refer to vehicles, buildings, etc. (mainly buildings) in the video footage, such as making signs on buildings to guide people.

[0035] Step S103: Obtain the first driving status information and the second driving status information of the vehicle at the start time and the end time, respectively;

[0036] Specifically, the first driving state information includes the vehicle's first heading angle, first acceleration, and first speed at the starting moment; the second driving state information includes the vehicle's second heading angle, second acceleration, and second speed at the ending moment. The first and second accelerations can be acquired using a gyroscope. The first and second speeds are calculated from the vehicle's own rotational speed or obtained from GPS information. The first and second heading angles can be determined by acquiring angles from GPS information.

[0037] Step S104: Based on the start time, the first driving state information, the end time, the second driving state information, and the latitude and longitude of the vehicle when the current frame is acquired, calculate the latitude and longitude of the vehicle when the next frame is acquired; wherein, the start time is the start timestamp of rendering, and the end time is the end timestamp of rendering. The start timestamp and end timestamp are recorded in the vehicle system and can be accurate to nanoseconds, thereby further reducing calculation errors, improving accuracy, and ensuring that the model fits the actual situation in real time.

[0038] Specifically, the process of obtaining the vehicle's latitude and longitude in the next frame is as follows: First, the distance traveled by the vehicle is calculated based on the acceleration, velocity, and time consumed. Then, the position of the next latitude and longitude is deduced based on the distance and direction. The acceleration and velocity can be obtained from the first and second driving state information, and the time consumed can be determined by calculating the time difference between the start and end times.

[0039] Step S105: Render the next frame based on the calculated latitude and longitude of the vehicle when acquiring the next frame.

[0040] like Figure 2 As shown, in one implementable manner, step S104 includes:

[0041] Step S1041: Calculate the rendering time based on the start time and the end time;

[0042] In a specific embodiment, if the start time is T1, the end time is T2, and the rendering time is T, then T = T2 - T1.

[0043] Step S1042: Calculate the distance the vehicle travels during the rendering of the current frame based on the first acceleration, the second acceleration, the first velocity, the second velocity, and the rendering time.

[0044] In a specific embodiment, if the first acceleration is ΔV1, the second acceleration is ΔV2, the first velocity is V1, the second velocity is V2, and the rendering time is T, then the distance S traveled by the vehicle during the rendering of the current frame can be calculated based on the above parameters.

[0045] Step S1043: Calculate the latitude and longitude of the vehicle when acquiring the next frame based on the distance traveled by the vehicle, the first direction angle, and the second direction angle.

[0046] This solution calculates the vehicle's latitude and longitude at the time of acquiring the next frame based on the first acceleration, second acceleration, first velocity, second velocity, rendering time, first direction angle, and second direction angle. This improves the accuracy of acquiring the vehicle's latitude and longitude, effectively reduces calculation errors, and increases precision. This latitude and longitude is then used as a benchmark in modeling calculations to ensure the model closely matches the actual implementation. In one feasible approach, the position information of the target object relative to the vehicle at the time of acquiring the current frame is calculated based on the vehicle's latitude and longitude, the target object's latitude and longitude in the current frame, and the first direction angle. The current frame is then rendered based on this position information to form a navigation model on the current frame screen. The position information includes the coordinate data of the target object in a three-dimensional coordinate system with the center of the vehicle's front as the origin.

[0047] In this solution, the latitude and longitude of the target object in each frame can be obtained by searching for points of interest in the map data. The points of interest in the map data contain the geographical location information of the target object.

[0048] In this solution, the current frame is rendered based on the latitude and longitude of the vehicle, the latitude and longitude of the target object in the current frame, and the first direction angle to ensure the accuracy of the navigation model formed on the current frame screen. This ensures that the model fits the reality in real time, so that the rendered object can always accurately fit the reality when the vehicle equipped with the AR navigation system is driving.

[0049] In one feasible approach, when the positioning device updates the vehicle's positioning information, the latitude and longitude of the vehicle used for rendering the current frame are calibrated based on the updated positioning information.

[0050] In this scheme, when the positioning device updates the vehicle's positioning information, the latitude and longitude of the vehicle used for rendering the current frame are calibrated based on the updated positioning information. This allows the positioning device to automatically select the actual value detected by the positioning device when updating the vehicle's positioning information, thereby further improving the accuracy of obtaining the vehicle's latitude and longitude, effectively reducing calculation errors, and improving precision. This latitude and longitude is then used as a benchmark in the modeling calculation to ensure that the model fits the reality in real time.

[0051] Example 2

[0052] like Figure 3 As shown, this embodiment discloses an AR navigation image rendering device, which is used to render real-world images in video frames. The device includes:

[0053] Video frame acquisition module 1 is used to acquire the current frame in a video frame.

[0054] Specifically, the current frame is the frame that is being rendered.

[0055] Rendering module 2 is used to render the current frame and record the start and end times of the rendering.

[0056] Specifically, the rendered objects refer to vehicles, buildings, etc. (mainly buildings) in the video footage, such as making signs on buildings to guide people.

[0057] Data acquisition module 3 is used to acquire the first driving status information and the second driving status information of the vehicle at the start time and the end time, respectively;

[0058] Specifically, the first driving state information includes the vehicle's first heading angle, first acceleration, and first speed at the starting moment; the second driving state information includes the vehicle's second heading angle, second acceleration, and second speed at the ending moment. The first and second accelerations can be acquired using a gyroscope. The first and second speeds are calculated from the vehicle's own rotational speed or obtained from GPS information. The first and second heading angles can be determined by acquiring angles from GPS information.

[0059] Processing module 4 is used to calculate the latitude and longitude of the vehicle when acquiring the next frame based on the start time, the first driving state information, the end time, the second driving state information, and the vehicle's latitude and longitude at the time of acquiring the current frame; wherein, the start time is the start timestamp of rendering, and the end time is the end timestamp of rendering. The start timestamp and end timestamp are recorded in the vehicle system and can be accurate to nanoseconds, thereby further reducing calculation errors, improving accuracy, and ensuring that the model fits the actual situation in real time.

[0060] Specifically, the process of obtaining the vehicle's latitude and longitude in the next frame is as follows: First, the distance traveled by the vehicle is calculated based on the acceleration, velocity, and time consumed. Then, the position of the next latitude and longitude is deduced based on the distance and direction. The acceleration and velocity can be obtained from the first and second driving state information, and the time consumed can be determined by calculating the time difference between the start and end times.

[0061] The rendering module 2 is also used to render the next frame based on the calculated latitude and longitude of the vehicle when the next frame is acquired.

[0062] like Figure 4 As shown, in one implementable embodiment, processing module 4 includes:

[0063] The first calculation unit 41 is used to calculate the rendering time based on the start time and the end time;

[0064] In a specific embodiment, if the start time is T1, the end time is T2, and the rendering time is T, then T = T2 - T1.

[0065] The second calculation unit 42 is used to calculate the distance traveled by the vehicle during the rendering of the current frame based on the first acceleration, the second acceleration, the first velocity, the second velocity and the rendering time.

[0066] In a specific embodiment, if the first acceleration is ΔV1, the second acceleration is ΔV2, the first velocity is V1, the second velocity is V2, and the rendering time is T, then the distance S traveled by the vehicle during the rendering of the current frame can be calculated based on the above parameters.

[0067] The third calculation unit 43 is used to calculate the latitude and longitude of the vehicle when acquiring the next frame based on the distance traveled by the vehicle, the first direction angle, and the second direction angle.

[0068] This solution calculates the vehicle's latitude and longitude at the time of acquiring the next frame based on the first acceleration, second acceleration, first velocity, second velocity, rendering time, first direction angle, and second direction angle. This improves the accuracy of acquiring the vehicle's latitude and longitude, effectively reduces calculation errors, and increases precision. This latitude and longitude is then used as a benchmark in the modeling calculations to ensure the model closely matches reality in real time. In one feasible implementation, the rendering module 2 is further configured to calculate the position information of the target object relative to the vehicle at the time of acquiring the current frame based on the vehicle's latitude and longitude, the target object's latitude and longitude in the current frame, and the first direction angle. Based on this position information, the current frame is rendered to form a navigation model on the current frame screen. The position information includes the coordinate data of the target object in a three-dimensional coordinate system with the center of the vehicle's front as the origin.

[0069] In this solution, the latitude and longitude of the target object in each frame can be obtained by searching for points of interest in the map data. The points of interest in the map data contain the geographical location information of the target object.

[0070] In this solution, the current frame is rendered based on the latitude and longitude of the vehicle, the latitude and longitude of the target object in the current frame, and the first direction angle to ensure the accuracy of the navigation model formed on the current frame screen. This ensures that the model fits the reality in real time, so that the rendered object can always accurately fit the reality when the vehicle equipped with the AR navigation system is driving.

[0071] In one possible implementation, the apparatus further includes a calibration module 5 for calibrating the latitude and longitude of the vehicle used for rendering the current frame based on the updated positioning information when the positioning device updates the vehicle's positioning information.

[0072] In this scheme, when the positioning device updates the vehicle's positioning information, the latitude and longitude of the vehicle used for rendering the current frame are calibrated based on the updated positioning information. This allows the positioning device to automatically select the actual value detected by the positioning device when updating the vehicle's positioning information, thereby further improving the accuracy of obtaining the vehicle's latitude and longitude, effectively reducing calculation errors, and improving precision. This latitude and longitude is then used as a benchmark in the modeling calculation to ensure that the model fits the reality in real time.

[0073] Example 3

[0074] Figure 5This is a schematic diagram of an electronic device provided in Embodiment 3 of the present invention. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it implements the AR navigation image rendering method provided in Embodiment 1. Figure 5 The electronic device 50 shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of the present invention.

[0075] like Figure 5 As shown, the electronic device 50 can be manifested in the form of a general-purpose computing device, such as a server device. The components of the electronic device 50 may include, but are not limited to: at least one processor 51, at least one memory 52, and a bus 53 connecting different system components (including memory 52 and processor 51).

[0076] Bus 53 includes a data bus, an address bus, and a control bus.

[0077] The memory 52 may include volatile memory, such as random access memory (RAM) 521 and / or cache memory 522, and may further include read-only memory (ROM) 523.

[0078] The memory 52 may also include a program / utility 525 having a set (at least one) of program modules 524, including but not limited to: an operating system, one or more application programs, other program modules, and program data, each or some combination of these examples may include an implementation of a network environment.

[0079] The processor 51 executes various functional applications and data processing by running computer programs stored in the memory 52, such as the AR navigation image rendering method provided in Embodiment 1 of the present invention.

[0080] Electronic device 50 can also communicate with one or more external devices 54 (e.g., keyboard, pointing device, etc.). This communication can be performed via input / output (I / O) interface 55. Furthermore, the model-generated device 50 can also communicate with one or more networks (e.g., local area network (LAN), wide area network (WAN), and / or public networks, such as the Internet) via network adapter 56. As shown, network adapter 56 communicates with other modules of the model-generated device 50 via bus 53. It should be understood that, although not shown in the figure, other hardware and / or software modules can be used in conjunction with the model-generated device 50, including but not limited to: microcode, device drivers, redundant processors, external disk drive arrays, RAID (disk array) systems, tape drives, and data backup storage systems.

[0081] It should be noted that although several units / modules or sub-units / modules of the electronic device have been mentioned in the detailed description above, this division is merely exemplary and not mandatory. In fact, according to embodiments of the present invention, the features and functions of two or more units / modules described above can be embodied in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided and embodied by multiple units / modules.

[0082] Example 4

[0083] This embodiment provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the AR navigation image rendering method provided in Embodiment 1.

[0084] The readable storage medium may be more specifically adopted, including but not limited to: portable disk, hard disk, random access memory, read-only memory, erasable programmable read-only memory, optical storage device, magnetic storage device, or any suitable combination thereof.

[0085] In a possible implementation, the present invention can also be implemented as a program product comprising program code, which, when the program product is run on a terminal device, is used to cause the terminal device to execute the AR navigation image rendering method provided in Embodiment 1.

[0086] The program code for executing the present invention can be written in any combination of one or more programming languages. The program code can be executed entirely on the user device, partially on the user device, as a standalone software package, partially on the user device and partially on a remote device, or entirely on a remote device.

[0087] While specific embodiments of the present invention have been described above, those skilled in the art should understand that these are merely illustrative examples, and the scope of protection of the present invention is defined by the appended claims. Those skilled in the art can make various changes or modifications to these embodiments without departing from the principles and essence of the present invention, but all such changes and modifications fall within the scope of protection of the present invention.

Claims

1. An AR navigation image rendering method, characterized in that, The method is used to render real-world images in video frames, and the method includes: obtaining the current frame in the video frame; Render the current frame and record the start and end times of the rendering. The first driving status information and the second driving status information of the vehicle are obtained at the start time and the end time, respectively. The latitude and longitude of the vehicle at the time of acquiring the next frame are calculated based on the start time, the first driving state information, the end time, the second driving state information, and the latitude and longitude of the vehicle at the time of acquiring the current frame. The next frame is rendered based on the calculated latitude and longitude of the vehicle when the next frame is acquired.

2. The method according to claim 1, wherein the first driving state information includes the first heading angle, first acceleration, and first speed of the vehicle at the starting time; and the second driving state information includes the second heading angle, second acceleration, and second speed of the vehicle at the ending time.

3. The method according to claim 2, wherein the step of calculating the latitude and longitude of the vehicle at the time of acquiring the next frame based on the first driving state information, the second driving state information, and the latitude and longitude of the vehicle at the time of acquiring the current frame includes: Calculate the rendering time based on the start time and the end time; The distance traveled by the vehicle during the rendering of the current frame is calculated based on the first acceleration, the second acceleration, the first velocity, the second velocity, and the rendering time. The latitude and longitude of the vehicle are calculated based on the distance traveled by the vehicle, the first direction angle, and the second direction angle when the next frame is acquired.

4. The method according to claim 2, further comprising: Based on the latitude and longitude of the vehicle at the time of acquiring the current frame, the latitude and longitude of the target object in the current frame, and the first direction angle, the position information of the target object relative to the vehicle at the time of acquiring the current frame is calculated, and the current frame is rendered based on the position information of the target object relative to the vehicle at the time of acquiring the current frame to form a navigation model on the current frame screen, wherein the position information includes the coordinate data of the target object in a three-dimensional coordinate system with the center of the front of the vehicle as the origin.

5. The method according to claim 1, wherein the start time is the start timestamp of rendering and the end time is the end timestamp of rendering.

6. The method according to claim 1, further comprising: When the positioning device updates the vehicle's positioning information, it calibrates the latitude and longitude of the vehicle used to render the current frame based on the updated positioning information.

7. The method according to claim 4, further comprising: The latitude and longitude of the target object in each frame are obtained by searching for points of interest in the map data, wherein the points of interest in the map data contain the geographical location information of the target object.

8. The method according to claim 2, wherein the first acceleration and the second acceleration are acquired by a gyroscope.

9. An AR navigation image rendering device, characterized in that, The device is used to render real-world images in video frames, and the device includes: The video frame acquisition module is used to acquire the current frame in a video frame. The rendering module is used to render the current frame and record the start and end times of the rendering. The data acquisition module is used to acquire the first driving state information and the second driving state information of the vehicle at the start time and the end time, respectively. The processing module is used to calculate the latitude and longitude of the vehicle when the current frame is acquired based on the start time, the first driving state information, the end time, the second driving state information, and the latitude and longitude of the vehicle when the next frame is acquired. The rendering module is also used to render the next frame based on the calculated latitude and longitude of the vehicle when the next frame is acquired.

10. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

Images