Navigation method and device, electronic equipment and computer readable storage medium

Abstract

The embodiment of the invention discloses a navigation method and device, electronic equipment and a computer readable storage medium, and relates to the technical field of augmented reality. The method is applied to augmented reality equipment and comprises the following steps: acquiring initial position information of a user; acquiring a destination; generating navigation information according to the starting position information and the destination of the user; the navigation information comprises a navigation path, annotation information and / or prompt information; generating a virtual navigation element according to the navigation information, and displaying the virtual navigation element in a real environment; when it is monitored that the user indicates the to-be-confirmed direction, feeding back whether the to-be-confirmed direction is matched with the correct direction in the navigation information to the user; and when the distance between the latest position information of the user and the destination is smaller than the target distance, displaying a destination prompt virtual element. Therefore, the scheme can realize efficient navigation.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of extended reality technology, and in particular, to a navigation method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] Related navigation technologies can display navigation paths and inform users of the next action route based on their real-time positions. However, the information provided by related navigation technologies to users is less. For example, it can only prompt users that they have deviated from the navigation route after they have deviated, but cannot help users find the correct route in advance; when there are many buildings or stores around the destination, it is also difficult for users to quickly find the destination based on the navigation path, resulting in low navigation efficiency and poor user experience. Summary of the Invention

[0003] Embodiments of the present application provide a navigation method, apparatus, electronic device, and computer-readable storage medium, which can achieve efficient navigation.

[0004] In a first aspect, an embodiment of the present application provides a navigation method applied to an extended reality device, including: Obtaining the starting position information of the user; Obtaining the destination; Generating navigation information according to the starting position information of the user and the destination; the navigation information includes a navigation path, annotation information, and / or prompt information; Generating virtual navigation elements according to the navigation information and displaying the virtual navigation elements in the real environment; When it is detected that the user indicates a direction to be confirmed, feeding back to the user whether the direction to be confirmed matches the correct direction in the navigation information; When the distance between the latest position information of the user and the destination is less than a target distance, displaying a destination prompt virtual element.

[0005] In one of the embodiments, the step of, when it is detected that the user indicates a direction to be confirmed, feeding back to the user whether the direction to be confirmed matches the correct direction in the navigation information includes: When it is detected that the user indicates a direction to be confirmed, obtaining the real-time position information of the user; When the real-time position information of the user indicates that the user is at an intersection, determining the correct direction corresponding to the intersection in the navigation information; Determining whether the direction to be confirmed matches the correct direction; When the direction to be confirmed matches the correct direction, feeding back to the user that the direction is correct.

[0006] In one embodiment, when it is detected that the user indicates a direction to be confirmed, feeding back to the user whether the direction to be confirmed matches the correct direction in the navigation information includes: When detecting the direction indicated by the user's hand, obtaining the user's line of sight direction; When the user's line of sight direction matches the direction indicated by the hand, confirming the direction indicated by the user's hand as the direction to be confirmed; Feeding back to the user whether the direction to be confirmed matches the correct direction in the navigation information.

[0007] In one embodiment, the method further includes: When the direction to be confirmed does not match the correct direction, determining the deviation between the direction to be confirmed and the correct direction; Obtaining the environmental information corresponding to the deviation; Generating a direction correction prompt according to the deviation and the environmental information corresponding to the deviation; Feeding back the direction correction prompt to the user.

[0008] In one embodiment, when the distance between the user's latest position information and the destination is less than the target distance, displaying a destination prompt virtual element includes: When the distance between the user's latest position information and the destination is less than the target distance, obtaining the surrounding environmental information; Determining whether the destination is included in the user's target field of view according to the surrounding environmental information and the user's latest position information; When the destination is included in the user's target field of view, displaying the destination prompt virtual element.

[0009] In one embodiment, determining whether the destination is included in the user's target field of view according to the surrounding environmental information and the user's latest position information includes: Obtaining the straight line area between the destination and the user's latest position information; Determining whether there are obstacles in the straight line area according to the surrounding environmental information; Determining whether the destination is included in the user's target field of view according to whether there are obstacles in the straight line area.

[0010] In one embodiment, the starting position information includes floor information; the extended reality device includes a barometer; Obtaining the user's starting position information includes: Measure the altitude of the user's starting position through the barometer; Obtain the altitude range corresponding to each floor of the building where the user's starting position is located; Determine the floor information of the user according to the altitude of the user's starting position and the altitude ranges corresponding to each floor.

[0011] In one embodiment, the method further includes: Obtain the line-of-sight direction of the user; When it is monitored that the duration for which the user's line-of-sight direction is not on the virtual navigation element exceeds a duration threshold, prompt the user in a target manner; the target manner includes: voice prompt and / or vibration prompt.

[0012] In a second aspect, an embodiment of the present application provides a navigation device applied to an extended reality device. The device includes: A position acquisition module, configured to acquire the starting position information of the user; A destination acquisition module, configured to acquire a destination; A generation module, configured to generate navigation information according to the user's starting position information and the destination; the navigation information includes a navigation path, annotation information, and / or prompt information; A navigation display module, configured to generate virtual navigation elements according to the navigation information and display the virtual navigation elements in the real environment; A feedback module, configured to, when it is monitored that the user indicates a direction to be confirmed, feedback to the user whether the direction to be confirmed matches the correct direction in the navigation information; A destination display module, configured to display a destination prompt virtual element when the distance between the user's latest position information and the destination is less than a target distance.

[0013] In a third aspect, an embodiment of the present application further provides an electronic device. The electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the computer program is executed by the processor, the steps in the above navigation method are implemented.

[0014] ] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium. A computer program is stored on the computer-readable storage medium. When the computer program is executed by the processor, the steps in the above navigation method are implemented.

[0015] In a fifth aspect, an embodiment of the present application further provides a computer program product or a computer program. The computer program product or the computer program includes computer instructions, and the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in the various alternative implementations described in the embodiments of the present application.

[0016] The embodiments of the present application have the following beneficial effects: Navigation information can be generated according to the starting position information and the destination of the user, and then virtual navigation elements corresponding to the navigation information can be displayed in the real environment. The navigation information includes various information such as a navigation path, annotation information, and / or prompt information. Therefore, multiple pieces of information can be provided to the user to achieve efficient navigation. The user can indicate the direction to be confirmed, and then it can be fed back to the user whether the direction to be confirmed matches the correct direction in the navigation information, so as to help the user find the correct route in advance, rather than prompting the user with an error after yawing. When the user arrives near the destination and the distance from the destination is less than the target distance, the destination prompt virtual element can be displayed to help the user quickly find the destination. Generally speaking, the navigation efficiency is improved and the user experience is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present application, the drawings required for the description of the embodiments will be briefly introduced below. Obviously, the drawings in the following description are only some embodiments of the present invention, and those skilled in the art can obtain other drawings according to these drawings without creative efforts.

[0018] Figure 1 It is a schematic diagram of the steps of a navigation method provided by an embodiment of the present application; Figure 2 It is a schematic diagram of the structure of a navigation device provided by an embodiment of the present application; Figure 3 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0019] The technical solutions in the present application will be clearly and completely described below with reference to the drawings in the present application. Obviously, the described embodiments are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative efforts fall within the protection scope of the present invention.

[0020] In one embodiment, as Figure 1As shown, a navigation method is provided. Although the logical order is shown in the step schematic diagram, in some cases, the steps shown or described can be executed in an order different from that shown in the drawings. Specifically, this navigation method can be applied to extended reality devices.

[0021] Extended reality includes virtual reality, augmented reality, and mixed reality. Extended reality devices include mobile handheld terminals such as mobile phones and tablets, include head-up displays such as those in cars, and include near-eye display terminals worn on the human head such as glasses-shaped displays and helmet-shaped displays. In the embodiments of the present application, a glasses-shaped display of the optical see-through type, that is, an extended reality glasses, is taken as an example to illustrate the hardware structure of the extended reality device.

[0022] The extended reality glasses include a frame body, that is, temple arms and a spectacle frame, to support the extended reality glasses to be worn on the human head; include an optical display module mainly composed of a micro display screen, an optical lens, and a waveguide plate to display virtual content to the user; include an audio module mainly composed of a microphone and a speaker to collect and play sounds; include sensors mainly composed of a camera, a gyroscope, a barometer, an infrared light emitter and receiver, etc. to collect information data related to the human body, the glasses body, and the external environment; include an integrated processor with a microcontroller unit (MCU) or a central processing unit (CPU) as the core to perform data processing and calculation; include a circuit board, flexible or rigid, to connect other electronic components to form an electronic circuit system, and the electronic circuit system is generally placed in the inner cavities of the glasses frame and temple arms; and may also include a battery power supply.

[0023] The following will be described in detail respectively. It should be noted that the description order of the following embodiments does not limit the priority order of the embodiments.

[0024] According to Figure 1 the navigation method shown, this method at least includes steps S110 to S160, which are introduced in detail as follows: In step S110, the starting position information of the user is obtained.

[0025] An extended reality device can determine the user's position in a two-dimensional plane through the Global Positioning System (GPS), wireless network, Bluetooth, ultra-wideband, and / or Inertial Measurement Unit (IMU), etc., and measure the altitude of the position where the user is located through a barometer. Based on the user's position in the two-dimensional plane and the altitude, the user's three-dimensional space position (including real-time position, starting position, etc.) can be determined. The user's starting position information is the real-time position information when the user starts using the navigation function.

[0026] Optionally, the obtained user position information (including real-time position information, starting position information, etc.) may include floor information. The extended reality device may include a barometer. Through the barometer, the altitude of the position where the user is located (including real-time position, starting position, etc.) can be measured.

[0027] When the position where the user is located is inside a building (the user's position information coincides with the building's position information), the altitude ranges corresponding to each floor of the building where the user is located can be obtained. Among them, the building data of the building can be obtained, and based on the building data of the building, the altitude ranges corresponding to each floor of the building can be determined. When the building data of the building cannot be obtained, the default floor height corresponding to the building type of the building can be obtained, and based on the default floor height, the altitude ranges corresponding to each floor of the building can be determined. Based on the altitude of the position where the user is located and the altitude ranges corresponding to each floor, the user's floor information can be determined.

[0028] For example, when the altitude where the building is located is 0 and the building data indicates that each floor is 3 meters high, then the altitude range corresponding to the first floor can be determined to be 0 - 3 meters, and the altitude range corresponding to the second floor can be determined to be 3 - 6 meters. When the altitude of the position where the user is located measured by the barometer is 4 meters, it can be determined that the user is on the second floor.

[0029] In this way, the user's floor information can be accurately determined, so as to accurately determine the user's position, and then based on the accurate position of the user, precise navigation for the user can be realized.

[0030] Based on the user's position in the two-dimensional plane and the floor where the user is located, the user's three-dimensional space position (including real-time position, starting position, etc.) in the indoor area can be determined.

[0031] In step S120, the destination is obtained.

[0032] When the navigation function is turned on, the extended reality device can accept the destination input by the user. Among them, the user can input the destination on the virtual keyboard or input the destination by voice.

[0033] Optionally, the extended reality device can accept the destination sent by other terminal devices.

[0034] In step S130, navigation information is generated according to the starting position information of the user and the destination.

[0035] Taking the user's starting position information as the starting point and the destination as the end point, one or more sets of navigation information can be intelligently generated. Both the user's starting position information and the destination are three-dimensional spatial addresses, and the navigation path in the generated navigation information is also three-dimensional, which can include multiple directions such as forward, backward, left, right, up, and down. Therefore, this navigation method can be applied to indoor navigation, floor navigation, etc.

[0036] Navigation in related technologies is often two-dimensional. For example, when the user arrives near the destination, the map shows that the destination is in front of the user. However, after the user walks forward, it may be found that there is no destination in front. The user may have to try many times and look around before finding that the destination is actually underground or upstairs, etc.

[0037] However, the navigation path in the embodiment of the present application is three-dimensional, which can guide the user to go upstairs and downstairs, etc., so as to accurately reach the destination. For example, if the destination is upstairs from the user's current position and the stairs to go upstairs are on the right, the navigation information can guide the user to go to the right and then go upstairs to reach the destination. In this way, it can also help the user quickly find the stairs.

[0038] The navigation information can include a navigation path, annotation information, and / or prompt information. The annotation information can be used to annotate buildings, stores, and / or roads, etc. encountered on the navigation path. For example, when moving along the navigation path, there is a convenience store 10 meters ahead of the current position. Then, annotation information can be generated according to the name, main business category, and / or business hours of the convenience store, etc., and then the annotation information can be displayed in the direction where the convenience store is located later.

[0039] The prompt information can include various prompt information. One kind of prompt information can be to prompt the user's traveling direction. For example, multiple consecutive arrows can be displayed on the ground (displaying virtual arrow elements on the real ground) to prompt the user's traveling direction. One kind of prompt information can be to prompt obstacles. For example, the user can be prompted with a prominent color that there are obstacles, so that the user can discover and avoid the obstacles to ensure the safety of the user. One kind of prompt information can be to prompt the user to pay attention to the navigation information. One kind of prompt information can be to prompt the location of the destination where the user is.

[0040] Each set of navigation information may include a navigation path. Each set of navigation information may include annotation information and / or prompt information corresponding to the navigation path. Among them, the annotation information and / or prompt information in the navigation information may be generated in real time as the user's position changes and the environment changes during navigation.

[0041] When generating multiple sets of navigation information, multiple navigation paths may be presented to the user, and the user selects one of the navigation paths. Subsequently, navigation is performed for the user according to the navigation information corresponding to the selected navigation path.

[0042] In step S140, virtual navigation elements are generated according to the navigation information, and the virtual navigation elements are displayed in the real environment.

[0043] According to the navigation information, virtual navigation elements can be generated using a virtual display screen. The virtual navigation elements can have different forms. For example, the navigation path can be an arrow, and the obstacle prompt information can be a red exclamation mark, etc.

[0044] The virtual display screen is generated in front of the user's eyes after the light emitted by the optical engine of the physical display screen undergoes multiple propagations and conversions. The virtual display screen is located in the user's real field of view and can be used to superimpose and display various virtual elements.

[0045] Through the virtual display screen, the virtual navigation elements can be displayed in the real environment to achieve the effect of combining virtual and real. By displaying the virtual navigation elements, it can help the user intuitively obtain the navigation information.

[0046] For example, if there is a convenience store 10 meters in front of the current position, the annotation information of the convenience store can be displayed at the location of the convenience store.

[0047] Optionally, according to the eye tracker, the convergence angle formed by the user's two eyeballs can be obtained. Based on the convergence angle, the direction and depth of the user's gaze can be calculated, so as to accurately determine the specific position of the user's gaze. For the annotation information, it can be displayed only when the user gazes at the object that needs to be annotated, avoiding the display of too many invalid annotations and affecting the user experience.

[0048] In step S150, when it is detected that the user indicates the direction to be confirmed, feedback is given to the user on whether the direction to be confirmed matches the correct direction in the navigation information.

[0049] During the navigation process, the user can indicate a direction to be confirmed. When it is detected that the user indicates a direction to be confirmed, it can be determined whether the direction to be confirmed matches the correct direction in the navigation information, and then the matching result can be fed back to the user. Among them, the correct direction in the navigation information refers to the direction corresponding to the user's current position in the navigation path. The matching of two directions means that the angular difference between the two directions is less than the angular threshold, and the angular threshold can be set according to actual needs. Among them, the matching result can be fed back to the user through voice and / or vision.

[0050] When the direction to be confirmed matches the correct direction in the navigation information, the feedback matching result can be that the direction is correct, or that the user should move in that direction, etc. When the direction to be confirmed does not match the correct direction in the navigation information, the feedback matching result can be that the direction is incorrect, or the correct direction can be fed back, etc. The feedback matching result is used to help the user move towards the correct direction in the navigation information.

[0051] In one embodiment, the user indicating the direction to be confirmed can be that the user points a finger at a certain direction.

[0052] In one embodiment, the user indicating the direction to be confirmed can be that the user points a finger at a certain direction and asks whether the direction is correct; when the camera of the extended reality device captures that the user points a finger at a certain direction, and the voice collection device of the extended reality device captures the voice of the user asking whether the direction is correct, it can be determined that the user's indication of the direction to be confirmed is detected, and the direction pointed by the user's finger can be determined as the direction to be confirmed.

[0053] In another embodiment, when it is detected that the user indicates the direction to be confirmed, feeding back whether the direction to be confirmed matches the correct direction in the navigation information to the user can include: when it is detected that the user's hand indicates a direction, obtaining the user's line of sight direction; when the user's line of sight direction matches the direction indicated by the hand, confirming the direction indicated by the user's hand as the direction to be confirmed; feeding back whether the direction to be confirmed matches the correct direction in the navigation information to the user.

[0054] When it is detected that the user's hand indicates a certain direction (the duration of the user's hand pointing at a certain direction reaches the preset duration), the user's line of sight direction can be obtained through a high-precision eye tracker, and it can be determined whether the user's line of sight direction matches the direction indicated by the user's hand.

[0055] When the user's line of sight direction matches the direction indicated by the user's hand, it is determined that the user indicates the direction to be confirmed, and the direction indicated by the user's hand is determined as the direction to be confirmed. When it is detected that the user indicates the direction to be confirmed, it can be determined whether the direction to be confirmed matches the correct direction in the navigation information, and then the matching result can be fed back to the user.

[0056] In the embodiments of the present application, considering that in real life, to confirm whether a certain direction is correct, one usually looks in that direction. Therefore, the user's line-of-sight direction and hand-indication direction are comprehensively considered to determine whether the user indicates the direction to be confirmed. Compared with simply judging based on the user's hand movements, it can more accurately determine whether the user is indicating the direction to be confirmed and avoid making wrong responses to the user's actions.

[0057] In step S160, when the distance between the user's latest position information and the destination is less than the target distance, a destination prompt virtual element is displayed.

[0058] In related navigation technologies, because there are many stores, after the user arrives near the destination, it sometimes takes a long time to find the destination. For example, in a food street, the types of adjacent stores are usually similar, and the decoration styles of each store are also similar. Sometimes, even if the target store is within 10 meters, it still takes repeated searching to find the target store.

[0059] To solve this technical problem and achieve better navigation, the user's position information can be obtained in real time. When the user's latest position information indicates that the user has arrived near the destination, that is, when the distance between the user's latest position information and the destination is less than the target distance, a destination prompt virtual element can be displayed. Among them, the target distance can be set according to actual needs.

[0060] The destination prompt virtual element can be used to prominently display the destination, and the user can easily see the destination prompt virtual element, so as to quickly determine the destination according to the destination prompt virtual element.

[0061] By adopting the technical solution of the embodiments of the present application, navigation information can be generated based on the user's starting position information and the destination, and then the virtual navigation elements corresponding to the navigation information can be displayed in the real environment. The navigation information includes various information such as navigation paths, annotation information, and / or prompt information. Therefore, multiple information can be provided to the user to achieve efficient navigation. The user can indicate the direction to be confirmed, and then it can be fed back to the user whether the direction to be confirmed matches the correct direction in the navigation information, so as to help the user find the correct route in advance instead of prompting the user with an error after deviation. When the user arrives near the destination and the distance from the destination is less than the target distance, the destination prompt virtual element can be displayed to help the user quickly find the destination. Generally speaking, the navigation efficiency is improved and the user experience is enhanced.

[0062] Based on the above technical solution, as an embodiment, when it is detected that the user indicates a direction to be confirmed, feeding back to the user whether the direction to be confirmed matches the correct direction in the navigation information may include: when it is detected that the user indicates a direction to be confirmed, obtaining the real-time position information of the user; when the real-time position information of the user indicates that the user is at an intersection, determining the correct direction corresponding to the intersection in the navigation information; determining whether the direction to be confirmed matches the correct direction; when the direction to be confirmed matches the correct direction, feeding back to the user that the direction is correct.

[0063] An intersection refers to an area with multiple direction options. For example, the entrance of a staircase can be regarded as an intersection. At the entrance of a staircase, one can choose to go upstairs, downstairs, or stay on the current floor. When there is only one road, usually one only needs to proceed along that road.

[0064] Therefore, when the user is not currently at an intersection, if it is detected that the user indicates a direction to be confirmed, it can be considered that this action of the user is not actually an indication of a direction to be confirmed, so there is no need to judge whether the direction is correct, nor to feed back to the user.

[0065] Specifically, when it is detected that the user indicates a direction to be confirmed through hand movements or the like, the real-time position information of the user can be obtained. According to the real-time position information of the user, it can be judged whether the user is currently at an intersection. When the user is currently at an intersection, the correct direction corresponding to the intersection can be determined according to the navigation information.

[0066] It can be judged whether the direction to be confirmed indicated by the user is the correct direction corresponding to the intersection. When the direction to be confirmed indicated by the user matches the correct direction corresponding to the intersection, it can be fed back to the user that the direction is correct. When the direction to be confirmed indicated by the user does not match the correct direction corresponding to the intersection, it can be fed back to the user that the direction is incorrect. By feeding back to the user, it can help the user proceed in the correct direction.

[0067] Adopting the technical solution of the embodiment of the present application to judge whether the directions match only at intersections, and thus feeding back to the user whether the direction is correct, can avoid the situation where the user makes a wrong reaction when actually not indicating a direction to be confirmed.

[0068] Based on the above technical solution, as an embodiment, the navigation method may include: when the direction to be confirmed does not match the correct direction, determining the deviation between the direction to be confirmed and the correct direction; obtaining the environmental information corresponding to the deviation; generating a direction correction prompt according to the deviation and the environmental information corresponding to the deviation; feeding back the direction correction prompt to the user.

[0069] When the direction to be confirmed indicated by the user does not match the correct direction in the navigation information, the smaller angle between the direction to be confirmed and the correct direction can be determined as the deviation between the direction to be confirmed and the correct direction. The surrounding environment information can be obtained according to the map information or by the extended reality device capturing the current environment image. The environment information corresponding to the smaller angle between the direction to be confirmed and the correct direction is determined as the environment information corresponding to the deviation. The extended reality device can have an artificial intelligence model built-in or call the artificial intelligence model of the cloud service. The artificial intelligence model can generate a direction correction prompt according to the deviation and the environment information corresponding to the deviation, and feedback the direction correction prompt to the user. Among them, the direction correction prompt can be fed back to the user through voice and / or vision.

[0070] For example, when the user is at an intersection, the direction to be confirmed indicated by the user is due east, while the correct direction is due north, and there is a shopping mall in the angle between due east and due north. Then it can be determined that the environment information corresponding to the deviation includes the shopping mall, and further a direction correction prompt can be generated as "Deviating from the direction you indicated towards the shopping mall can lead you to the correct direction".

[0071] Another example, if there is a flower shop in the direction to be confirmed indicated by the user and a restaurant in the correct direction, and there is a convenience store between the flower shop and the restaurant, then a direction correction prompt can be generated as "The correct direction is the direction where the restaurant is located next to the convenience store next to the flower shop indicated".

[0072] Adopting the technical solution of the embodiment of the present application, when the direction to be confirmed indicated by the user does not match the correct direction, the direction correction prompt generated by combining the environment information can intuitively, clearly and explicitly indicate the correct direction to the user. Through the direction correction prompt, even a person with a poor sense of direction can quickly and accurately find the correct direction.

[0073] Based on the above technical solution, as an embodiment, the displaying the destination prompt virtual element when the distance between the user's latest position information and the destination is less than the target distance may include: when the distance between the user's latest position information and the destination is less than the target distance, obtaining the surrounding environment information; determining whether the destination is included in the user's target visual field according to the surrounding environment information and the user's latest position information; and displaying the destination prompt virtual element when the destination is included in the user's target visual field.

[0074] Optionally, the distance between the user's latest position information and the destination can be the navigation distance. Compared with the straight-line distance, the navigation distance can more accurately measure the distance the user needs to travel to reach the destination.

[0075] It may be to obtain the surrounding environment information when the distance between the user's latest location information and the destination is less than the target distance. Among them, the extended reality device can collect the surrounding environment images, so as to obtain the surrounding environment information.

[0076] Analyze and process the surrounding environment information. When the surrounding environment images collected by the extended reality device include the image of the destination, it can be determined that the destination is included in the user's target field of view; when the surrounding environment images collected by the extended reality device do not include the image of the destination, it can be determined that the destination is not included in the user's target field of view.

[0077] It may be to display the destination prompt virtual element only when the destination is included in the user's target field of view. The destination prompt virtual element can be used to prominently display the destination, and the user can easily see the destination prompt virtual element, so as to quickly determine the destination according to the destination prompt virtual element. For example, the destination prompt virtual element can be to prompt the destination with a prominent green sign.

[0078] Optionally, the destination prompt virtual element can be displayed in the direction where the destination is located, or the destination prompt virtual element can be displayed at the location where the destination is located, so that when the user sees the destination prompt virtual element, they can quickly know the location of the destination.

[0079] For example, if the destination is a convenience store, the destination prompt virtual element can be displayed on the facade of the convenience store through extended reality technology, so that when the user sees the destination prompt virtual element, they can clearly know that the facade where the destination prompt virtual element is located is the destination.

[0080] When the destination is not included in the user's target field of view, a destination guiding element can be displayed. The destination guiding element can guide the user to the destination. The destination guiding element can be similar to the navigation path, but can have a different display format from the navigation path. For example, different elements or different colors can be used to make the user pay more attention to the destination guiding element when approaching the destination, so that they can quickly find the destination under the indication of the destination guiding element.

[0081] Adopting the technical solution of the embodiment of the present application, when the user arrives near the destination, it can be determined whether the destination is included in the user's target field of view, so as to display the destination prompt virtual element or the destination guiding element, thereby helping the user quickly find the destination, improving the navigation efficiency, and improving the user experience.

[0082] Based on the above technical solution, as an embodiment, determining whether the destination is included in the user's target field of view according to the surrounding environment information and the user's latest location information may include: obtaining a straight-line area between the destination and the user's latest location information; determining whether there are obstacles in the straight-line area according to the surrounding environment information; and determining whether the destination is included in the user's target field of view according to whether there are obstacles in the straight-line area.

[0083] The straight-line area between the destination and the user's latest location information can be determined according to map information. This straight-line area is three-dimensional. For example, if the destination is on the left side of the second floor and the user's latest location information is on the right side of the first floor, the corresponding straight-line area is the area corresponding to the straight line from the left side of the second floor to the right side of the first floor. The width of the straight-line area can be pre-configured or determined according to the size of the destination.

[0084] According to the surrounding environment information, it can be determined whether there are obstacles in the straight-line area. When there are objects in the straight-line area that can completely block the destination or block a preset proportion of the destination, it is determined that there are obstacles in the straight-line area; otherwise, it is determined that there are no obstacles in the straight-line area.

[0085] When there are obstacles in the straight-line area, it is determined that the destination is not included in the user's target field of view. When there are no obstacles in the straight-line area, it is determined that the destination is included in the user's target field of view.

[0086] By adopting the technical solution of the embodiment of the present application, the straight-line area can be determined and whether there are obstacles in the straight-line area can be determined only through map information, without collecting images of the surrounding environment through the camera of the extended reality device, thereby saving the computing resources of the extended reality device.

[0087] Based on the above technical solution, as an embodiment, the navigation method may further include: obtaining the user's line of sight direction; when the duration of the user's line of sight direction not being on the virtual navigation element exceeds a duration threshold, prompting the user in a target manner; the target manner includes: voice prompt and / or vibration prompt.

[0088] To avoid the user from yawing, the user's line of sight direction can be obtained through a high-precision eye tracker. When it is monitored that the user's line of sight direction is not on the virtual navigation element for a long time, that is, the duration of the user's line of sight direction not being on the virtual navigation element exceeds the duration threshold, the user can be prompted to pay attention to the virtual navigation element. Among them, the duration threshold can be set according to actual needs.

[0089] Optionally, the convergence angle formed by the two eyeballs of the user can be obtained according to the eye tracker. Based on the convergence angle, the direction and depth of the user's gaze can be calculated, so as to accurately determine the specific position of the user's gaze.

[0090] When the position where the user is gazing is not on the virtual navigation element for a long time, the user can be prompted to pay attention to the virtual navigation element.

[0091] Optionally, the user can be prompted in a target manner, and the target manner can include but is not limited to any one or more of the following: voice prompt, vibration prompt, and visual prompt. The voice prompt can be to output a prompt voice. For example, a voice prompt of "Please pay attention to the navigation information" can be output.

[0092] The extended reality device worn by the user can be provided with a vibration device, and the vibration device can emit vibrations to give the user a vibration prompt to prompt the user to pay attention to the navigation information.

[0093] The visual prompt can be to display a virtual element at the position where the user is currently gazing to prompt the user to pay attention to the navigation information. For example, a text prompt of "Please pay attention to the navigation information" can be displayed at the position where the user is currently gazing.

[0094] Optionally, the selection of the target manner can be determined according to the user's current attention. For example, when it is detected that the user's line of sight deviates from the navigation path, it is possible to preferentially prompt the user through voice rather than vision. For example, if the user is currently listening to music, a vibration or visual reminder can be used to avoid the coincidence of the prompt voice and the song, making it difficult for the user to notice the prompt voice.

[0095] By adopting the technical solution of the embodiment of the present application, when the user does not pay attention to the navigation information for a long time, the user can be prompted to pay attention to the navigation information, thereby avoiding the user from deviating from the course, so as to solve the problem of the user deviating from the course through prompting before the user deviates from the course.

[0096] To facilitate better implementation of the navigation method of the present application, the present application also provides a navigation device based on the above navigation method. The meanings of the terms are the same as those in the above navigation method, and the specific implementation details can be referred to the description in the method embodiment.

[0097] Please refer to Figure 2 , Figure 2 is a schematic structural diagram of the navigation device provided by the embodiment of the present application. The navigation device is applied to an extended reality device, and the navigation device includes: A position acquisition module 201, configured to acquire the starting position information of the user; A destination acquisition module 202, configured to acquire the destination; A generation module 203, configured to generate navigation information according to the starting position information of the user and the destination; the navigation information includes a navigation path, annotation information, and / or prompt information; A navigation display module 204, configured to generate virtual navigation elements according to the navigation information and display the virtual navigation elements in the real environment; A feedback module 205, configured to, when it is detected that the user indicates a direction to be confirmed, feedback to the user whether the direction to be confirmed matches the correct direction in the navigation information; A destination display module 206, configured to display a destination prompt virtual element when the distance between the latest position information of the user and the destination is less than a target distance.

[0098] In one embodiment, the feedback module 205 includes: A real-time position acquisition unit, configured to acquire the real-time position information of the user when it is detected that the user indicates a direction to be confirmed; A direction determination unit, configured to determine the correct direction corresponding to the intersection in the navigation information when the real-time position information of the user indicates that the user is at an intersection; A matching unit, configured to determine whether the direction to be confirmed matches the correct direction; A feedback unit, configured to feedback to the user that the direction is correct when the direction to be confirmed matches the correct direction.

[0099] In one embodiment, the feedback module 205 includes: A line-of-sight direction acquisition unit, configured to acquire the line-of-sight direction of the user when it is detected that the user's hand indicates a direction; A direction confirmation unit, configured to confirm the direction indicated by the user's hand as the direction to be confirmed when the line-of-sight direction of the user matches the direction indicated by the hand; A matching feedback unit, configured to feedback to the user whether the direction to be confirmed matches the correct direction in the navigation information.

[0100] In one embodiment, the device further includes: A deviation determination module, configured to determine the deviation between the direction to be confirmed and the correct direction when the direction to be confirmed does not match the correct direction; An environment acquisition module, configured to acquire the environment information corresponding to the deviation; A correction generation module, configured to generate a direction correction prompt according to the deviation and the environment information corresponding to the deviation; A correction feedback module, configured to feedback the direction correction prompt to the user.

[0101] In one embodiment, the destination display module 206 includes: A surrounding information acquisition unit, configured to acquire surrounding environment information when the distance between the latest position information of the user and the destination is less than the target distance; A field of view determination unit, configured to determine whether the destination is included in the target field of view of the user according to the surrounding environment information and the latest position information of the user; A prompt display unit, configured to display the destination prompt virtual element when the destination is included in the target field of view of the user.

[0102] In one embodiment, the field of view determination unit includes: A region acquisition subunit, configured to acquire a straight-line region between the destination and the latest position information of the user; An occlusion determination subunit, configured to determine whether there is an occluder in the straight-line region according to the surrounding environment information; A destination determination subunit, configured to determine whether the destination is included in the target field of view of the user according to whether there is an occluder in the straight-line region.

[0103] In one embodiment, the starting position information includes floor information; the extended reality device includes a barometer; the position acquisition module 201 includes: A measurement unit, configured to measure the altitude of the starting position of the user through the barometer; A range acquisition unit, configured to acquire the altitude ranges corresponding to each floor of the building where the starting position of the user is located; A floor determination unit, configured to determine the floor information of the user according to the altitude of the starting position of the user and the altitude ranges corresponding to each floor.

[0104] In one embodiment, the device further includes: A line-of-sight direction acquisition module, configured to acquire the line-of-sight direction of the user; A target prompt module, configured to prompt the user in a target manner when it is monitored that the duration for which the line-of-sight direction of the user is not on the virtual navigation element exceeds a duration threshold; the target manner includes: voice prompt and / or vibration prompt.

[0105] By adopting the technical solution of the embodiment of the present application, navigation information can be generated based on the starting position information and destination of the user, and thus the virtual navigation elements corresponding to the navigation information can be displayed in the real environment. The navigation information includes various information such as navigation paths, annotation information, and / or prompt information, etc. Therefore, various information can be provided for the user to achieve efficient navigation; the user can indicate the direction to be confirmed, and thus it can be fed back to the user whether the direction to be confirmed matches the correct direction in the navigation information, so as to help the user find the correct route in advance, rather than prompting the user with an error after yawing; when the user arrives near the destination and the distance from the destination is less than the target distance, the destination prompt virtual element can be displayed, thereby helping the user quickly find the destination; generally speaking, the navigation efficiency is improved and the user experience is improved.

[0106] For the specific limitations of the navigation device, reference can be made to the limitations on the navigation method in the above text, which will not be elaborated here. Each module in the above navigation device can be implemented in whole or in part by software, hardware, and their combinations. The above-mentioned modules can be embedded in the processor in the computer device in hardware form or independent of it, or stored in the memory in the computer device in software form, so that the processor can call and execute the operations corresponding to the above-mentioned modules.

[0107] In addition, the present application also provides an electronic device, such as Figure 3 shown, which shows the structural schematic diagram of the electronic device involved in the present application. Specifically: The electronic device may include a processor 301 with one or more processing cores and a memory 302 with one or more computer-readable storage media and other components. Those skilled in the art can understand that Figure 3 the structure of the electronic device shown in does not constitute a limitation on the electronic device, and it may include more or fewer components than shown, or combine certain components, or have different component arrangements. Among them: The processor 301 is the control center of the electronic device, connecting various parts of the entire electronic device through various interfaces and lines. By running or executing the software programs and / or modules stored in the memory 302, and calling the data stored in the memory 302, it executes various functions of the electronic device and processes data, thereby monitoring the electronic device as a whole. Optionally, the processor 301 may include one or more processing cores; preferably, the processor 301 may integrate an application processor and a modulation and demodulation processor. Among them, the application processor mainly processes the operating system, user interface, and application programs, etc., and the modulation and demodulation processor mainly processes wireless communication. It can be understood that the above modulation and demodulation processor may not be integrated into the processor 301.

[0108] The memory 302 can be used to store software programs and modules. The processor 301 executes various functional applications and data processing by running the software programs and modules stored in the memory 302. The memory 302 mainly includes a program storage area and a data storage area. Among them, the program storage area can store the operating system, application programs required for at least one function (such as the sound playback function, the image playback function, etc.); the data storage area can store data created according to the use of the electronic device. In addition, the memory 302 can include high-speed random access memory, and can also include non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage devices. Correspondingly, the memory 302 can also include a memory controller to provide the processor 301 with access to the memory 302.

[0109] In one embodiment, the electronic device further includes a power supply 303 for powering each component. Preferably, the power supply 303 can be logically connected to the processor 301 through a power management system, so as to implement functions such as management of charging, discharging, and power consumption management through the power management system. The power supply 303 can also include any components such as one or more DC or AC power supplies, a recharge system, a power device debugging circuit, a power converter or inverter, and a power status indicator.

[0110] In one embodiment, the electronic device may further include an input unit 304, which can be used to receive input digital or character information, and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function controls.

[0111] Although not shown, the electronic device may further include a display unit, etc., which will not be elaborated here. Specifically, in this embodiment, the processor 301 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 302 according to the following instructions, and the processor 301 will run the application programs stored in the memory 302, so as to implement the steps in any of the navigation methods provided in the embodiments of the present application.

[0112] When the specific electronic device is a wearable optically see-through smart glasses, in addition to the above structure, it at least further includes a glasses body frame, an optical display component, an electronic circuit component, a sensor, etc. The sensors built into the glasses include a heart rate monitor, a blood glucose detector, a microphone, a camera, and / or an eye tracker.

[0113] Those skilled in the art can understand, Figure 3The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the electronic device to which the solution of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0114] In one embodiment, an electronic device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method described in any embodiment of the present application is implemented.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the method described in any embodiment of the present application is implemented.

[0116] In some embodiments, a computer program product is also proposed, including a computer program or instructions, which implements the method described in any embodiment of the present application when executed by a processor.

[0117] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0118] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0119] To this end, the present application provides a computer-readable storage medium, on which a computer program is stored. The computer program can be loaded by a processor to execute the steps in any navigation method provided in the present application.

[0120] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0121] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0122] Since the instructions stored in the computer-readable storage medium can execute the steps in any navigation method provided in the present application, the beneficial effects that can be achieved by any navigation method provided in the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0123] Finally, it should also be noted that in this text, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or terminal device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal device. Without further limitation, an element defined by the statement "comprising an..." does not exclude the presence of additional identical elements in the process, method, article or terminal device comprising the said element.

[0124] The above has introduced in detail a navigation method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used in this text to elaborate on the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those skilled in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. A navigation method, characterized in that, Applied to an extended reality device, the method includes: Obtain the starting position information of the user; Obtain the destination; Generate navigation information based on the starting position information of the user and the destination; the navigation information includes a navigation path, annotation information, and / or prompt information; Generate a virtual navigation element based on the navigation information and display the virtual navigation element in the real environment; When it is detected that the user indicates a direction to be confirmed, feedback to the user whether the direction to be confirmed matches the correct direction in the navigation information; When the distance between the user's latest position information and the destination is less than a target distance, display a destination prompt virtual element.

2. The navigation method according to claim 1, wherein The step of, when it is detected that the user indicates a direction to be confirmed, feedback to the user whether the direction to be confirmed matches the correct direction in the navigation information includes: When it is detected that the user indicates a direction to be confirmed, obtain the real-time position information of the user; When the real-time position information of the user indicates that the user is at an intersection, determine the correct direction corresponding to the intersection in the navigation information; Determine whether the direction to be confirmed matches the correct direction; When the direction to be confirmed matches the correct direction, feedback to the user that the direction is correct.

3. The navigation method according to claim 1, characterized in that, The step of, when it is detected that the user indicates a direction to be confirmed, feedback to the user whether the direction to be confirmed matches the correct direction in the navigation information includes: When it is detected that the user's hand indicates a direction, obtain the user's line of sight direction; When the user's line of sight direction matches the direction indicated by the hand, confirm the direction indicated by the user's hand as the direction to be confirmed; Feedback to the user whether the direction to be confirmed matches the correct direction in the navigation information.

4. The navigation method according to claim 1, characterized in that, The navigation method further includes: When the direction to be confirmed does not match the correct direction, determine the deviation between the direction to be confirmed and the correct direction; Obtain the environmental information corresponding to the deviation; Generate a direction correction prompt based on the deviation and the environmental information corresponding to the deviation; Feedback the direction correction prompt to the user.

5. The navigation method according to claim 1, wherein, The step of, when the distance between the user's latest position information and the destination is less than a target distance, display a destination prompt virtual element includes: When the distance between the user's latest position information and the destination is less than the target distance, obtain the surrounding environmental information; Based on the surrounding environmental information and the user's latest position information, determine whether the destination is included in the user's target field of view; When the destination is included in the user's target field of view, display the destination prompt virtual element.

6. The navigation method according to claim 5, characterized in that The step of, based on the surrounding environmental information and the user's latest position information, determine whether the destination is included in the user's target field of view includes: Obtain the straight-line area between the destination and the user's latest position information; Based on the surrounding environmental information, determine whether there are obstacles in the straight-line area; Based on whether there are obstacles in the straight-line area, determine whether the destination is included in the user's target field of view.

7. The navigation method according to claim 1, characterized in that, The starting position information includes floor information; the extended reality device includes a barometer; The obtaining of the user's starting position information includes: Measuring the altitude of the user's starting position through the barometer; Obtaining the altitude ranges corresponding to each floor of the building where the user's starting position is located; Determining the user's floor information according to the altitude of the user's starting position and the altitude ranges corresponding to each floor.

8. The navigation method according to claim 1, characterized in that The navigation method further includes: Obtaining the user's line of sight direction; When the continuous duration during which the user's line of sight direction is not on the virtual navigation element exceeds a duration threshold, prompting the user in a target manner; the target manner includes: voice prompt and / or vibration prompt.

9. A navigation device, characterized in that, Applied to an extended reality device, the device includes: A position acquisition module for acquiring the user's starting position information; A destination acquisition module for acquiring a destination; A generation module for generating navigation information according to the user's starting position information and the destination; the navigation information includes a navigation path, annotation information, and / or prompt information; A navigation display module for generating virtual navigation elements according to the navigation information and displaying the virtual navigation elements in the real environment; A feedback module for, when it is detected that the user indicates a direction to be confirmed, feeding back whether the direction to be confirmed matches the correct direction in the navigation information to the user; A destination display module for displaying a destination prompt virtual element when the distance between the user's latest position information and the destination is less than a target distance.

10. An electronic device, characterized in that, Comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implements the steps in the navigation method according to any one of claims 1 to 8 when executing the computer program.

11. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, and when the computer program is executed by a processor, the steps in the navigation method according to any one of claims 1 to 8 are implemented.

Images