Ar navigation method and device, electronic device and computer storage medium

Abstract

Embodiments of the present application provide an AR navigation method and device, electronic equipment and computer storage medium. The AR navigation method comprises: obtaining attitude information of a device; obtaining image features of a real scene image, the real scene image being collected by a visual sensor integrated or externally connected to the device; determining whether a time for drawing a road-related navigation element is reached according to the image features and the attitude information, and if so, drawing the road-related navigation element on the real scene image based on positioning information of the device and navigation route data. The method has better navigation effect.

Description

Technical Field

[0001] This application relates to the field of navigation technology, and in particular to an AR navigation method, device, electronic device, and computer storage medium. Background Technology

[0002] In navigation scenarios, to make navigation guidance information more intuitive, existing technologies combine AR (Augmented Reality) with navigation guidance to form AR navigation. AR navigation uses integrated or external image acquisition devices to capture real-world images and draws navigation elements on the real-world images displayed on the device screen to guide the navigating object.

[0003] A problem with existing AR navigation is that the navigation elements drawn in the real-world image do not match the scene. For example, there may be no road in the real-world image, but road-related navigation elements are drawn. The inventors discovered that the reason for this problem is that existing AR navigation technology cannot accurately determine when to draw navigation elements. Summary of the Invention

[0004] In view of this, embodiments of this application provide an AR navigation scheme to at least partially solve the above-mentioned problems.

[0005] According to a first aspect of the embodiments of this application, an AR navigation method is provided, comprising: acquiring posture information of a device; acquiring image features of a real-scene image, wherein the real-scene image is acquired by a visual sensor integrated into or external to the device; determining, based on the image features and the posture information, whether an opportunity has been reached to draw road-related navigation elements; if so, drawing road-related navigation elements on the real-scene image based on the positioning information and navigation route data of the device.

[0006] According to a second aspect of the embodiments of this application, an AR navigation device is provided, comprising: a first acquisition module for acquiring posture information of the device; a second acquisition module for acquiring image features of a real-world image, wherein the real-world image is acquired by a vision sensor integrated into or external to the device; and a drawing module for determining, based on the image features and the posture information, whether the timing for drawing road-related navigation elements has been reached, and if so, drawing road-related navigation elements on the real-world image based on the positioning information and navigation route data of the device.

[0007] According to a third aspect of the present application, an electronic device is provided, comprising: a processor, a memory, a communication interface, and a communication bus, wherein the processor, the memory, and the communication interface communicate with each other via the communication bus; the memory is used to store at least one executable instruction, wherein the executable instruction causes the processor to perform an operation corresponding to the AR navigation method described in the first aspect.

[0008] According to a fourth aspect of the embodiments of this application, a computer storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the AR navigation method as described in the first aspect.

[0009] According to the AR navigation scheme provided in the embodiments of this application, during the navigation process, it is determined whether the timing for drawing road-related navigation elements has been reached based on the acquired device attitude information and the image features of the collected real-scene image. If the timing is reached, road-related navigation elements are drawn on the real-scene image based on the device's positioning information and navigation route data, thereby realizing the overlay of road-related navigation elements on the real-scene image containing the real environment to better guide navigation. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings.

[0011] Figure 1A This is a flowchart illustrating the steps of an AR navigation method according to Embodiment 1 of this application;

[0012] Figure 1B for Figure 1A A schematic diagram of a scenario example in the illustrated embodiment;

[0013] Figure 2A This is a flowchart illustrating the steps of an AR navigation method according to Embodiment 2 of this application;

[0014] Figure 2B for Figure 2A A schematic diagram of a pitch angle and roll angle in the embodiment shown;

[0015] Figure 3 This is a structural block diagram of an AR navigation device according to Embodiment 3 of this application;

[0016] Figure 4 This is a schematic diagram of the structure of an electronic device according to Embodiment 5 of this application. Detailed Implementation

[0017] To enable those skilled in the art to better understand the technical solutions in the embodiments of this application, the technical solutions in the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art should fall within the protection scope of the embodiments of this application.

[0018] The specific implementation of the embodiments of this application will be further described below with reference to the accompanying drawings.

[0019] Example 1

[0020] Reference Figure 1A The flowchart of the AR navigation method according to Embodiment 1 of this application is shown.

[0021] In this embodiment, the method includes the following steps:

[0022] Step S102: Obtain the device's attitude information.

[0023] The device can be a terminal device, such as a mobile phone, tablet, in-vehicle terminal, wearable terminal, etc. It can be mounted on vehicles or other means of transportation, or used by hand. In this embodiment, the method is applied to an AR (Augmented Reality) navigation scenario as an example for illustration.

[0024] Attitude information can be determined based on data collected by gyroscopes, accelerometers, and other devices on the equipment. Attitude information includes, but is not limited to, pitch and roll angles, and may also include yaw angle.

[0025] Step S104: Obtain the image features of the real-scene image, which is acquired by the vision sensor integrated into or external to the device.

[0026] When using an external vision sensor, the device can connect to the sensor via wired or wireless means. The acquired real-world images contain information about the device's environment.

[0027] Step S106: Based on the image features and the attitude information, determine whether the timing for drawing road-related navigation elements has been reached. If so, draw road-related navigation elements on the real-world image based on the device's positioning information and navigation route data.

[0028] In one feasible approach, a trained deep learning model can be used to process real-world images to obtain image features that can characterize information such as objects contained in the real-world image and the positional relationships between objects.

[0029] Based on image features, the objects contained in the environment in which the device is located can be determined, and thus it can be determined whether the device is in a road environment or in an environment outside the road environment (such as an indoor environment).

[0030] Attitude information can determine whether the device is mounted on a vehicle and is in a relatively stable state. If the attitude information indicates that it is in a stable state, and the image features indicate that the real-world image contains road-related features, then it is time to draw road-related navigation elements. These elements can then be drawn on the real-world image, making navigation guidance clearer and more intuitive.

[0031] To ensure the accuracy of the drawing, the precise drawing location can be determined from the real-world image based on the positioning information and navigation route data, and the corresponding road-related navigation elements can be drawn at the drawing location.

[0032] Road-related navigation elements can be determined based on the actions that need to be prompted. For example, if the navigable object needs to be guided to go straight along the current road, a straight-ahead arrow can be drawn as a road-related navigation element. If it needs to guide lane changes or turns, an arrow indicating merging into an adjacent lane or turning into another road can be drawn as a road-related navigation element. In addition to arrows, other directional signs, such as signs indicating slowing down, can also be used as road-related navigation elements; there are no restrictions on this.

[0033] Alternatively, if it's determined that the timing for drawing road-related navigation elements hasn't arrived, then instead of drawing them on the device's navigation interface, the captured real-world image can be displayed. This allows for dynamic and intelligent determination of whether the device is in road navigation mode based on the captured real-world image and the device's attitude information during navigation. When in road navigation mode, road-related navigation elements are drawn on the captured real-world image, providing more navigation information and making navigation guidance more intuitive and convenient, thereby improving navigation effectiveness.

[0034] When not in a road-related navigation state, road-related navigation elements are not drawn, thus avoiding drawing them at inappropriate times and causing distortion, in order to ensure navigation accuracy.

[0035] The implementation process of the method will be explained below using a specific use case:

[0036] Reference Figure 1B In this scenario, we will take AR navigation in a car using a mobile phone as an example for explanation:

[0037] The mobile phone can capture real-world images of its surroundings using its built-in visual sensors, and determine its attitude information using data collected from sensors such as the gyroscope and accelerometer on the phone.

[0038] If the phone's attitude is determined to be stable based on the attitude information, and the real-world image contains road elements, then the phone is determined to be in road navigation mode. Road-related navigation elements, such as road boundary lines and forward indicator signs, are then drawn on the acquired real-world image for display. Figure 1B As shown in Interface 1.

[0039] Alternatively, if the attitude information indicates that the phone is not in an stable state, or if the captured real-world image does not include road information, it means the phone is not in navigation mode, or is currently in an indoor environment or an environment without a road surface. In this case, drawing road-related navigation elements on the real-world image would cause navigation distortion and mismatch with the actual environment, affecting the navigation effect. In this situation, the captured real-world image can be left unprocessed, thus improving the intelligence of the drawing process. This makes the drawn road-related navigation elements more compatible with the image and better integrated into the real-world image, thereby improving the navigation effect. The real-world image without drawn road-related navigation elements can be directly displayed on the navigation interface, such as... Figure 1B As shown in Interface 2.

[0040] In this embodiment, during navigation, the system determines whether it is time to draw road-related navigation elements based on the acquired device attitude information and the image features of the collected real-world images. If so, road-related navigation elements are drawn on the real-world images based on the device's positioning information and navigation route data, thereby enabling the overlay of road-related navigation elements on real-world images containing the real environment for better navigation guidance.

[0041] The AR navigation method of this embodiment can be executed by any suitable electronic device with data processing capabilities, including but not limited to: servers, mobile terminals (such as mobile phones, PADs, etc.) and PCs.

[0042] Example 2

[0043] Reference Figure 2A The diagram shows a flowchart of the AR navigation method according to Embodiment 2 of this application.

[0044] In this embodiment, the method includes the following steps:

[0045] Step S202: Obtain the device's attitude information.

[0046] In one feasible approach, acquiring the attitude information of the device can be achieved by determining the attitude information of the device based on the measurement data output by the gyroscope and accelerometer integrated into the device.

[0047] like Figure 2B As shown, the attitude information includes at least the pitch angle and roll angle of the device. The pitch angle indicates the angle at which the device is raised or lowered, i.e., the angle between the device's x-axis and the ground (i.e., the angle at which the device rotates around the y-axis). The roll angle indicates the angle at which the device rotates around the x-axis.

[0048] Step S204: Obtain the image features of the real-scene image, which is acquired by the vision sensor integrated into or external to the device.

[0049] In one feasible approach, image features can be obtained by inputting the real-world image into a trained visual deep learning model to obtain the image features of the real-world image.

[0050] Visual deep learning models can be TensorFlow models. By inputting real-world images into the TensorFlow model, the TensorFlow model identifies the features and outputs the identified image features.

[0051] Step S206: Based on the image features and the attitude information, determine whether the timing for drawing road-related navigation elements has been reached. If so, draw road-related navigation elements on the real-world image based on the device's positioning information and navigation route data.

[0052] Based on the obtained image features, it can be determined whether the real-world image contains road surface features. If road surface features are present, it indicates that the vehicle carrying the equipment is likely on a road, rather than indoors. Combined with the equipment's attitude information, it can be determined whether the equipment is ready to render road-related elements.

[0053] In one example, the timing for drawing road-related navigation elements can be determined by: determining whether the image features contain road surface features, and determining whether the pitch angle matches the forward direction of the vehicle on which the device is located and whether the roll angle matches the ground; when the aforementioned determination results are both yes, it is determined to enter the road navigation state.

[0054] Road surface features include, but are not limited to, lane markings. Specifically, the pitch angle can be compared with the direction of travel determined by sensors such as accelerometers, and the roll angle can be compared with a horizontal plane or a reference plane determined by data collected from external sensors. If the difference is within a set range (which can be determined as needed), it is determined to match the ground; otherwise, it is not a match.

[0055] If the image features include road surface features, the pitch angle is consistent with the vehicle's heading (i.e., the direction of travel), and the roll angle is parallel to the ground, then it is determined that the time to draw has been reached.

[0056] Based on the device's location information, the device's location is determined. Then, based on the location and navigation route data, the road-related navigation elements that need to be drawn and their corresponding drawing positions are determined and drawn on the corresponding drawing positions in the real-world image, thereby improving the intuitiveness of navigation.

[0057] If any of the following conditions are met: the image features do not contain road features, the pitch angle does not match the direction of travel, or the roll angle does not match the ground, it indicates that the device is not being used for navigation. Therefore, road-related navigation elements are not drawn, which avoids drawing road-related navigation elements by mistake and improves accuracy.

[0058] Road-related navigation elements include signs, which are used to indicate directions and thus achieve navigation.

[0059] Optionally, in order to draw road-related navigation elements more accurately, the method further includes steps S208 to S210.

[0060] Step S208: Based on the device's positioning information and navigation route data, determine whether the device has deviated from its course or reached its destination.

[0061] Location information can be coordinates determined through methods such as satellite positioning and network positioning. Based on location information and navigation route data, it can be determined whether the device has reached its destination, or whether the location indicated by the location information deviates too far from the navigation route.

[0062] Step S210: If the route deviates or the destination is reached, stop drawing road-related navigation elements on the image.

[0063] In one feasible approach, it is not necessary to draw road-related navigation elements when a deviation is determined or the destination is reached, in order to avoid misguided navigation. This allows for timely and accurate determination of whether to terminate drawing to prevent mismatch between elements drawn on the real-world image and the real-world image.

[0064] In one feasible approach, steps S208 and S210 can be executed periodically. Alternatively, they can be executed after step S206 determines that the drawing opportunity has been reached, but before the road-related navigation elements are drawn. For example, after determining that the drawing opportunity has been reached, if it is determined based on the positioning information and navigation route data that there is no deviation and the destination has not been reached, then the road-related navigation elements are drawn on the real-world image based on the device's positioning information and navigation route data; or, if it is determined that there is a deviation or the destination has been reached, then the road-related navigation elements are not drawn.

[0065] The implementation process is explained below using a specific use case:

[0066] Taking navigation while the vehicle is in motion as an example, before the device is installed near the windshield of the vehicle, the device is in a handheld state. When the visual sensor has not captured a real-world image containing the road ahead, its image features do not include road surface features and do not match the posture of the navigation state according to the posture information. Therefore, road-related navigation elements are not drawn on the real-world image.

[0067] During the installation of the device near the windshield, the device's posture will shake significantly, and the visual sensor's field of view will be unstable. This will cause the device's posture information to not match the set posture information, and thus the road-related navigation elements will not be drawn.

[0068] After the device is installed near the windshield, if the vehicle is in a complex environment such as a parking lot, residential area, or internal road, the image features of the collected real-world images do not include road features. Therefore, although the device's attitude information matches the set attitude information, it is still determined not to draw road-related navigation elements.

[0069] When a vehicle travels onto a normal, open road, the image features of the captured real-world image contain road surface elements, and the device's attitude information matches the set attitude information, thus determining that it has entered the road navigation state. The system then draws road-related navigation elements in the real-world image and displays the drawn real-world image on the interface for navigation.

[0070] If the device is removed from near the windshield while the vehicle is in motion, such as when receiving a phone call, the device's posture information changes and does not match the set posture information. In this case, the system will exit the road navigation state and will not draw road-related navigation elements on the real-view image.

[0071] In this way, by using the real-world image of the scene ahead transmitted by the visual sensor, as well as the orientation information of the device, it can detect in real time whether there is a normal drivable road ahead, and thus determine whether to draw road-related navigation elements on the real-world image. This can intelligently prevent these elements from being drawn in non-road navigation states, avoiding the problem of AR effects not matching the real-world image.

[0072] When identifying drivable roads, a visual deep learning model that can run on the device is used to identify the captured real-world images to determine if road surface features are present, improving recognition accuracy. Based on the device's gyroscope and accelerometer, the device's attitude information (including pitch and roll angles) is determined, and this attitude information is used to accurately determine when to render, improving rendering accuracy. Furthermore, based on navigation route data and positioning information, it can be determined whether the device has deviated from its course or reached its destination. If it does, rendering can be stopped promptly, thus avoiding mismatches between rendered navigation elements and real-world image content.

[0073] Example 3

[0074] Reference Figure 3 The diagram shows a structural block diagram of the AR navigation device according to Embodiment 3 of this application.

[0075] According to this embodiment, the AR navigation device includes:

[0076] The first acquisition module 302 is used to acquire the device's attitude information;

[0077] The second acquisition module 304 is used to acquire image features of a real-scene image, wherein the real-scene image is acquired by a vision sensor integrated into or external to the device.

[0078] The drawing module 306 is used to determine whether the timing for drawing road-related navigation elements has been reached based on the image features and the posture information. If the timing is reached, the road-related navigation elements are drawn on the real-scene image based on the device's positioning information and navigation route data.

[0079] Optionally, the first acquisition module 302 is used to determine the attitude information of the device based on the measurement data output by the gyroscope and accelerometer integrated in the device.

[0080] Optionally, the second acquisition module 304 is used to input the real-scene image into a trained visual deep learning model to obtain the image features of the real-scene image.

[0081] Optionally, the attitude information includes the device's pitch angle and roll angle. The drawing module 306 is used to determine whether the image features contain road surface features when determining whether the timing for drawing road-related navigation elements has been reached based on the image features and the attitude information; whether the pitch angle matches the forward direction of the vehicle on which the device is located and whether the roll angle matches the ground; and when the aforementioned determination results are all yes, determine that the timing for drawing road-related navigation elements has been reached.

[0082] Optionally, the device further includes:

[0083] The first exit module 308 is used to determine the timing of non-compliance when any of the following conditions are met: the road surface features are not included, the pitch angle does not match the forward direction, or the roll angle does not match the ground.

[0084] Optionally, the device further includes:

[0085] The second exit module 310 is used to determine whether the device has deviated from its course or reached its destination based on the device's positioning information and navigation route data; if it has deviated from its course or reached its destination, it stops drawing road-related navigation elements in the image.

[0086] The navigation device of this embodiment is used to implement the corresponding AR navigation methods in the foregoing method embodiments and has the beneficial effects of the corresponding method embodiments, which will not be repeated here. Furthermore, the functional implementation of each module in the navigation device of this embodiment can be referred to the description of the corresponding part in the foregoing method embodiments, which will also not be repeated here.

[0087] Example 4

[0088] Reference Figure 4 The diagram shows a structural schematic of an electronic device according to Embodiment 4 of this application. The specific embodiments of this application do not limit the specific implementation of the electronic device.

[0089] like Figure 4 As shown, the electronic device may include: a processor 402, a communications interface 404, a memory 406, and a communications bus 408.

[0090] in:

[0091] The processor 402, communication interface 404, and memory 406 communicate with each other via communication bus 408.

[0092] Communication interface 404 is used to communicate with other electronic devices or servers.

[0093] The processor 402 is used to execute program 410, specifically to perform the relevant steps in the above-described AR navigation method embodiment.

[0094] Specifically, program 410 may include program code that includes computer operation instructions.

[0095] Processor 402 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application. The smart device includes one or more processors, which may be processors of the same type, such as one or more CPUs; or processors of different types, such as one or more CPUs and one or more ASICs.

[0096] Memory 406 is used to store program 410. Memory 406 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.

[0097] Specifically, program 410 can be used to cause processor 402 to perform the operation corresponding to the aforementioned method.

[0098] The specific implementation of each step in program 410 can be found in the corresponding steps and units described in the above-described AR navigation method embodiments, and will not be repeated here. Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the devices and modules described above can be referred to the corresponding process descriptions in the aforementioned method embodiments, and will not be repeated here.

[0099] This application also provides a computer program product, including computer instructions that instruct a computing device to perform an operation corresponding to any of the AR navigation methods in the above-described multiple method embodiments.

[0100] It should be noted that, depending on the implementation needs, the various components / steps described in the embodiments of this application can be broken down into more components / steps, or two or more components / steps or parts of the operation of components / steps can be combined into new components / steps to achieve the purpose of the embodiments of this application.

[0101] The methods described in the embodiments of this application can be implemented in hardware, firmware, or as software or computer code that can be stored in a recording medium (such as a CD-ROM, RAM, floppy disk, hard disk, or magneto-optical disk), or as computer code downloaded over a network that is originally stored in a remote recording medium or a non-transitory machine-readable medium and will be stored in a local recording medium. Thus, the methods described herein can be stored as software processing on a recording medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware (such as an ASIC or FPGA). It is understood that the computer, processor, microprocessor controller, or programmable hardware includes storage components (e.g., RAM, ROM, flash memory, etc.) capable of storing or receiving software or computer code that, when accessed and executed by the computer, processor, or hardware, implements the AR navigation methods described herein. Furthermore, when a general-purpose computer accesses code used to implement the AR navigation methods shown herein, the execution of the code transforms the general-purpose computer into a dedicated computer for executing the AR navigation methods shown herein.

[0102] Those skilled in the art will recognize that the units and method steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the embodiments of this application.

[0103] The above embodiments are only used to illustrate the embodiments of this application, and are not intended to limit the embodiments of this application. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the embodiments of this application. Therefore, all equivalent technical solutions also fall within the scope of the embodiments of this application, and the patent protection scope of the embodiments of this application should be defined by the claims.

Claims

1. An AR navigation method, comprising: Acquire the attitude information of the device, wherein the attitude information includes at least the pitch angle and roll angle of the device; Acquire image features of a real-world image, wherein the real-world image is acquired by a vision sensor integrated into or external to the device; Determine whether the image features contain road surface features; Determine whether the pitch angle matches the forward direction of the vehicle on which the equipment is located and whether the roll angle matches the ground. When all of the above judgment results are yes, the time is determined to draw road-related navigation elements; If this is achieved, then based on the device's positioning information and navigation route data, road-related navigation elements are drawn on the real-world image.

2. The method according to claim 1, wherein, The acquisition of the device's attitude information includes: The attitude information of the device is determined based on the measurement data output by the gyroscope and accelerometer integrated in the device.

3. The method according to claim 1, wherein, The image features acquired from the real-world image include: The real-scene image is input into a trained visual deep learning model to obtain the image features of the real-scene image.

4. The method according to claim 1, wherein, When any one of the following conditions is met—that the road surface features are not included, that the pitch angle does not match the direction of travel, or that the roll angle does not match the ground—the drawing timing is determined to be unsatisfactory.

5. The method according to any one of claims 1-4, wherein, The method further includes: Based on the device's positioning information and navigation route data, determine whether the device has deviated from its course or reached its destination; If the route deviates or the destination is reached, stop drawing road-related navigation elements in the image.

6. An AR navigation device, comprising: The first acquisition module is used to acquire the attitude information of the device, the attitude information including at least the pitch angle and roll angle of the device; The second acquisition module is used to acquire image features of a real-scene image, which is acquired by a vision sensor integrated into or external to the device. The drawing module is used to determine whether the image features include road surface features; to determine whether the pitch angle matches the forward direction of the vehicle on which the device is located and whether the roll angle matches the ground; when the aforementioned determination results are all yes, it determines the timing for drawing road-related navigation elements; if the timing is reached, it draws road-related navigation elements on the real-scene image based on the device's positioning information and navigation route data.

7. An electronic device, comprising: The processor, memory, communication interface, and communication bus are provided, wherein the processor, memory, and communication interface communicate with each other via the communication bus. The memory is used to store at least one executable instruction that causes the processor to perform the operation corresponding to the AR navigation method as described in any one of claims 1-5.

8. A computer storage medium having a computer program stored thereon, which, when executed by a processor, implements the AR navigation method as described in any one of claims 1-5.

9. A computer program product comprising computer instructions that instruct a computing device to perform an operation corresponding to any one of the AR navigation methods as described in claims 1-5.

Images