Augmented reality (AR) navigation method, electronic equipment and storage medium

Abstract

The invention provides an AR (Augmented Reality) navigation method, electronic equipment and a storage medium, relates to the technical field of navigation, and improves the precision of AR navigation by stopping a positioning function when determining that the current position of the electronic equipment is on an escalator in the AR navigation process. The method comprises the following steps: after an AR navigation application is started, starting a positioning function to determine the current position of the electronic equipment; and displaying a navigation route planning interface in response to the operation of inputting the destination. The navigation route planning interface comprises a navigation route determined based on the current position and the destination of the electronic equipment, in response to an operation of starting navigation, displaying an AR navigation interface comprising navigation guidance, and the navigation guidance is determined based on a first image collected by a camera and the current position of the electronic equipment; when it is determined that the current position of the electronic equipment is located on the escalator according to a second image collected by the camera, the positioning function is stopped, and navigation guidance is displayed on the AR navigation interface based on the navigation route.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of navigation technology, and in particular to an augmented reality (AR) navigation method, electronic device, and storage medium. Background Art

[0002] With the rapid development of Internet technology, map navigation has become a frequently used tool for people's daily travel. For example, when a user does not know how to reach an unfamiliar destination (such as a shopping mall, store, etc.), he or she usually turns on the map navigation function on the electronic device.

[0003] With the rapid development of hardware technology and visual algorithms, augmented reality (AR) technology is increasingly being applied to map navigation. Indoor AR navigation, in particular, has become increasingly popular, solving the problem of users finding their destination in complex or unfamiliar indoor environments, such as finding a store in a mall or their car in a garage. However, AR navigation still suffers from low navigation accuracy, influenced by factors such as location, position, and scene. Summary of the Invention

[0004] An embodiment of the present application provides an augmented reality (AR) navigation method, an electronic device, and a storage medium. By stopping the positioning function when determining that the current position of the electronic device is on an escalator during the AR navigation process, the accuracy of AR navigation in the escalator scenario is improved, thereby enhancing the user's travel experience.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, an augmented reality (AR) navigation method is provided, the method comprising:

[0007] After launching the AR navigation application, the positioning function is enabled to determine the current location of the electronic device; in response to the operation of inputting a destination, a navigation route planning interface is displayed. The navigation route planning interface includes a navigation route determined based on the current location of the electronic device and the destination. In response to the operation of starting navigation, the AR navigation interface is displayed, and the AR navigation interface includes navigation instructions. The navigation instructions are determined based on the first image captured by the camera and the current location of the electronic device. When it is determined that the current location of the electronic device is at a preset location based on the second image captured by the camera, the positioning function is stopped, and navigation instructions are displayed on the AR navigation interface based on the navigation route; the preset location includes an escalator.

[0008] This means that after AR navigation is activated, the positioning function is enabled. When the image captured by the camera determines that the electronic device is currently on the escalator, the positioning function is disabled. This avoids the problem of the AR navigation application updating to the wrong navigation route due to inaccurate positioning of the electronic device on the escalator, thereby improving the accuracy of AR navigation and enhancing the user's travel experience.

[0009] In some embodiments, the AR navigation method may further include:

[0010] The first prompt information is displayed, and the first prompt information is used to prompt that the current location is a preset location, and the navigation instructions are continuously displayed on the AR navigation interface.

[0011] That is, when the current location of the electronic device is on the escalator, the electronic device can display a prompt message in the interface to remind the user that the user is currently on the escalator.

[0012] In some embodiments, after stopping the positioning function, the AR navigation method may further include:

[0013] Obtain a third image captured by the camera; determine based on the third image that the current position of the electronic device is not at the preset position, turn on the positioning function, and after using the positioning function to determine the current position of the electronic device, display navigation instructions on the AR navigation interface.

[0014] That is to say, when the current position of the electronic device is on the escalator, the camera of the electronic device captures images in real time. When the electronic device determines that the electronic device has currently left the escalator based on the captured third image, the positioning function is turned on again. In the subsequent AR navigation process, after the current position of the electronic device is determined according to the positioning function, navigation instructions are displayed on the AR navigation interface, thereby improving the accuracy of AR navigation.

[0015] In some embodiments, the AR navigation method may further include:

[0016] A second prompt message is displayed, and the second prompt message is used to prompt that the preset position has been left.

[0017] That is to say, when the electronic device leaves the escalator, the electronic device can display a prompt message to prompt the user to currently leave the escalator. After the electronic device receives the triggering operation of the user determining to leave the escalator, the electronic device determines to have left the escalator.

[0018] In some embodiments, before determining that the current position of the electronic device is at a preset position, the AR navigation method may further include:

[0019] It is determined that the current position of the electronic device is in a target area, where the target area is an area corresponding to the escalator.

[0020] It can be understood that the electronic device can first determine that the current position is in the area corresponding to the escalator, and then determine whether the current position of the electronic device is on the escalator, thereby more accurately determining that the electronic device is on the escalator.

[0021] In some embodiments, determining that the current location of the electronic device is within the target area includes:

[0022] Acquire inertial measurement unit (IMU) data; determine the camera pose based on the second image and the IMU data; and determine that the electronic device's current position is within the target area when the electronic device's current position, as determined based on the camera pose, enters the escalator bounding box. The escalator bounding box is pre-generated based on the offline map and indicates the area corresponding to the escalator's position and size in the offline map.

[0023] Illustratively, the escalator bounding box may be a rectangular area, a square area, etc. The present application does not limit the shape of the escalator bounding box. For example, the escalator bounding box may also be a circular area, a diamond area, etc.

[0024] The camera pose refers to the position and orientation of the camera in three-dimensional space. After the electronic device determines its current position based on the camera pose, it determines that the current position enters the escalator bounding box, and thus determines that the current position of the electronic device is within the escalator area.

[0025] In some embodiments, when the escalator bounding box is a rectangular area or a square area, determining that the current position of the electronic device enters the escalator bounding box based on the camera pose includes:

[0026] Determine the minimum horizontal coordinate value, the maximum horizontal coordinate value, the minimum vertical coordinate value, and the maximum vertical coordinate value among the coordinate values ​​of the four vertices of the escalator bounding box;

[0027] When the horizontal coordinate value of the camera pose is less than the maximum horizontal coordinate value and greater than the minimum horizontal coordinate value, and the vertical coordinate value of the camera pose is less than the maximum vertical coordinate value and greater than the minimum vertical coordinate value, it is determined that the current position of the electronic device enters the escalator bounding box.

[0028] It can be understood that the electronic device can compare the camera posture with the coordinates of each vertex of the escalator bounding box in the camera coordinate system to determine whether the current position of the electronic device is within the escalator area, and can accurately determine whether the current position of the electronic device is within the escalator area.

[0029] In some embodiments, when the escalator bounding box is a rectangular area or a square area, determining that the current position of the electronic device enters the escalator bounding box based on the camera pose includes:

[0030] Determine the horizontal coordinate value and the vertical coordinate value of the center point of the escalator bounding box; when the difference between the horizontal coordinate value of the camera posture and the horizontal coordinate value of the center point is less than a first difference threshold, and the difference between the vertical coordinate value of the camera posture and the vertical coordinate value of the center point is less than a second difference threshold, determine that the current position of the electronic device enters the escalator bounding box.

[0031] It can be understood that the electronic device can determine whether the current position of the electronic device is within the escalator area based on the camera posture and the horizontal and vertical coordinate values ​​of the center point of the escalator bounding box.

[0032] In some embodiments, when the escalator bounding box is a circular area, determining that the current position of the electronic device enters the escalator bounding box based on the camera pose includes:

[0033] Determine the horizontal and vertical coordinate values ​​of the center point of the escalator bounding box; when the absolute value of the difference between the horizontal coordinate value of the camera posture and the horizontal coordinate value of the center point is less than a third difference threshold, and the absolute value of the difference between the vertical coordinate value of the camera posture and the vertical coordinate value of the center point is also less than the third difference threshold, determine that the current position of the electronic device enters the escalator bounding box, and the third difference threshold is the radius value of the circular area.

[0034] It can be understood that the electronic device can determine whether the current position of the electronic device is within the escalator area based on the difference between the coordinate value of the camera posture and the coordinate value of the center of the circular area, and the radius.

[0035] In some embodiments, the AR navigation method further includes:

[0036] An object detection algorithm is used to detect an object on the offline map to obtain a target object, which is an escalator; the position and size of the target object are determined; and a bounding box of the escalator is generated according to the position and size of the target object.

[0037] It can be understood that in the AR navigation process, the escalator bounding box can be used to identify and plan the best path for the user to reach the destination. The accuracy and completeness of the escalator bounding box are crucial to the accuracy of AR navigation applications.

[0038] In some embodiments, determining that the current location of the electronic device is at a preset location includes:

[0039] Performing target detection on the second image using a target detection algorithm to obtain a detection result;

[0040] If the detection result includes an escalator, it is determined that the current position of the electronic device is at a preset position.

[0041] It can be understood that in the process of an electronic device controlling an AR navigation application to perform AR navigation, the electronic device can perform target detection on the image captured by the camera. When the detection result includes an escalator, the electronic device determines that the current position is on the escalator, thereby improving the accuracy of escalator detection.

[0042] In some embodiments, after stopping the positioning function, the method further includes:

[0043] Stop initialization operations related to the positioning function.

[0044] It can be understood that after the electronic device determines that the current location is on the escalator and stops the positioning function, the electronic device does not perform any operations related to the positioning function, so as to avoid the problem that when performing initialization operations in dynamic environments such as escalators or elevators, certain detected data (for example, sensor-related data) are inaccurate, resulting in unsuccessful initialization operations.

[0045] In a second aspect, the present application provides an electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the navigation method described in any one of the first aspects above.

[0046] In a third aspect, the present application provides a computer-readable storage medium having a computer program / instruction stored thereon, which, when executed by a processor, implements the navigation method described in any one of the first aspects.

[0047] In a fourth aspect, the present application provides a computer program product, comprising a computer program / instruction, which, when executed by a processor, implements the navigation method described in any one of the first aspects.

[0048] It can be understood that the electronic device described in the second aspect, the computer-readable storage medium described in the third aspect, and the computer program product described in the fourth aspect are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 An example navigation diagram provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of the structure of a navigation system provided in an embodiment of the present application;

[0051] Figure 3 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0052] Figure 4A software structure schematic diagram of an electronic device provided for an embodiment of the present application;

[0053] Figure 5 An AR navigation example diagram provided for an embodiment of the present application;

[0054] Figure 6 A flowchart of a navigation method provided for an embodiment of the present application;

[0055] Figure 7 An example diagram for determining the position between a mobile phone and an escalator area provided for an embodiment of the present application;

[0056] Figure 8 A flowchart of a target detection method provided for an embodiment of the present application;

[0057] Figure 9 A network model structure diagram for target detection provided for an embodiment of the present application;

[0058] Figure 10 An AR navigation interface example diagram provided for an embodiment of the present application;

[0059] Figure 11 Another AR navigation interface example diagram provided for an embodiment of the present application;

[0060] Figure 12 Still another AR navigation interface example diagram provided for an embodiment of the present application;

[0061] Figure 13 A structure schematic diagram of an electronic device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. In the description of the embodiments of the present application, unless otherwise specified, “ / ” represents or, for example, A / B can represent A or B; in this document, “and / or” only describes the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which means that there are three cases of A alone, A and B together, and B alone.

[0063] Hereinafter, the terms “first” and “second” are only used for description purposes, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Therefore, the features defined with “first” and “second” can explicitly or implicitly include one or more features. In the description of the embodiments of the present application, unless otherwise specified, the meaning of “multiple” is two or more.

[0064] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0065] Currently, in the existing AR navigation software, during the navigation process in elevator or escalator scenarios, the elevator or escalator's own motion state has a certain impact on the acceleration and angular velocity detected by the electronic device's inertial measurement unit (IMU), resulting in positioning failure or large positioning drift during the AR navigation's online real-time positioning process.

[0066] An IMU is a sensor device used to detect the acceleration, angular velocity, and magnetic field of a device. An IMU can include an accelerometer, a gyroscope, a magnetic sensor, and more.

[0067] Positioning drift refers to the phenomenon in which the deviation between the device's location data provided by the AR navigation positioning system and the device's actual location gradually increases. This deviation may cause the positioning system's accuracy to decrease.

[0068] It can be understood that since the elevator or escalator itself moves at a certain acceleration during operation, when the user stands on the elevator or escalator, the accelerometer in the IMU of the electronic device performing AR navigation will detect additional acceleration (i.e., the acceleration of the elevator or escalator). If the elevator or escalator is rising, the accelerometer will detect an upward acceleration; if the elevator or escalator is descending, the accelerometer will detect a downward acceleration. This may be misinterpreted by the IMU as the movement of the electronic device itself, thereby affecting the accuracy of positioning and navigation.

[0069] Furthermore, the movement of the escalator can affect the accuracy of the angular velocity detected by the IMU's gyroscope. For example, when the escalator turns, the electronic device's IMU will detect additional angular velocity, causing the IMU to misjudge the device's actual orientation.

[0070] To accurately measure the motion of electronic devices, IMUs typically require a reference frame, assuming they are stationary relative to the ground. However, in AR navigation scenarios where electronic devices are located in elevators or escalators, this assumption no longer holds due to the movement of the elevators or escalators themselves. This can cause the IMUs of electronic devices to be unable to accurately detect their motion, leading to positioning failures or significant drift during online real-time positioning of electronic devices on elevators or escalators.

[0071] Furthermore, because AR navigation typically relies on global positioning system (GPS) signals to determine the device's location, when an elevator or escalator is located indoors, the indoor environment or building structure may block the GPS signal, resulting in inaccurate positioning. Alternatively, objects surrounding the elevator or escalator may reflect the GPS signal, causing multipath effects that further increase positioning errors.

[0072] Because electronic devices may fail to or inaccurately position themselves during AR navigation in elevators or escalators, when AR navigation software provides route guidance based on the device's real-time location, users may receive incorrect navigation instructions, leading to problems such as getting lost, taking the wrong path, or missing their destination, thereby reducing the accuracy of AR navigation.

[0073] For example, Figure 1 As shown, after the user enters the destination (e.g., a store on the 3rd floor of a shopping mall) on the AR navigation software, the phone displays the navigation route planned based on the current location and the location of the destination. Figure 1 As shown, during the AR navigation process, the mobile phone can use the AR navigation application to guide the user to walk to the escalator on the first floor, take the escalator to the second floor, and then guide the user to walk to the escalator on the second floor. After taking the escalator to the third floor of the mall, the navigation task is completed after walking to the destination on the third floor of the mall. During the AR navigation process, when the user is on the escalator, the IMU of the mobile phone cannot accurately detect the motion state of the mobile phone, resulting in positioning failure or large positioning drift during the online real-time positioning process of the escalator's AR navigation. For example, the actual location of the mobile phone is already on the escalator from the first floor to the second floor, but the mobile phone detects that the positioning result is on other floors (for example, the positioning position is on the 3rd floor). At this time, the AR navigation application of the mobile phone may switch to the wrong navigation route based on the currently detected positioning result, resulting in the inability to accurately and quickly navigate to the destination, reducing the user's travel experience.

[0074] To this end, an embodiment of the present application provides an AR navigation method, in which, after launching an AR navigation application, an electronic device activates a positioning function to determine the current location of the electronic device. When it is determined that the current location of the electronic device is on an escalator, the positioning function is stopped and navigation instructions are displayed on the AR navigation interface based on a pre-planned navigation route. This is until it is determined that the electronic device has left the escalator, at which point the positioning function is activated again. This avoids the problem of inaccurate positioning information obtained when the current location of the electronic device is on an escalator, which results in the replanning of the navigation route and the resulting incorrect navigation route, thereby improving the accuracy of AR navigation.

[0075] It can be understood that the embodiment of the present application provides an application scenario of an AR navigation method, which can be applied to shopping mall navigation scenarios, airport terminal navigation scenarios, factory navigation scenarios, exhibition hall navigation scenarios, hotel navigation scenarios, hospital navigation scenarios, and parking lot navigation scenarios, etc., where the navigation routes include elevator routes and / or escalator routes.

[0076] It should be explained that the application scenarios of the AR navigation method of the embodiment of the present application are not limited to indoor scenarios, but are also applicable to outdoor scenarios where the navigation routes include elevator routes and / or escalator routes.

[0077] The AR navigation method provided in the embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0078] The AR navigation method provided in the embodiment of the present application can be applied to a navigation system, which is used to plan and navigate a user's travel route, and the user's travel route includes an elevator route or an escalator route.

[0079] In some embodiments, as Figure 2 As shown, a navigation system 200 provided in an embodiment of the present application includes an electronic device 210 and a server 220. The server 220 can provide AR navigation information of a travel route to the electronic device 210, and the electronic device 210 can navigate the travel route based on the AR navigation information of the travel route provided by the server 220.

[0080] For example, suppose that an electronic device navigates to a store on a certain floor in a shopping mall through AR navigation software (that is, the navigation destination is located indoors). In some embodiments, the server 220 may store indoor map data for multiple buildings. For example, the tester can collect indoor 3D data of each building, and then model the indoor 3D data of each building to obtain indoor map data for each building; and store the indoor map data of each building in the server 220 (such as a cloud server). Among them, the server 220 can open an access interface to the electronic device 210 so that the electronic device 210 can obtain the indoor map data of the building where the destination is located through the access interface. During the AR navigation process, after the electronic device 210 receives the indoor map data of the building where the destination is located from the server, the electronic device 210 generates a navigation route based on the current position of the electronic device 210 and the indoor map data of the building where the destination is located, so that the electronic device 210 displays navigation instructions on the AR navigation interface based on the navigation route.

[0081] For example, the electronic device 210 in the navigation system 200 can be any device with AR navigation capabilities. For example, the device can be a mobile phone, a smart watch, a wristband, smart glasses, a tablet computer, a personal computer (PC), a personal digital assistant (PDA), a netbook, a wearable electronic device, an AR device, a virtual reality (VR) device, etc. The embodiments of the present application do not impose any restrictions on the specific form of the electronic device.

[0082] The server 220 in the above-mentioned navigation system 200 can be a server of an AR navigation application (application, APP) on the electronic device 210; the navigation application can be a third-party navigation application or a navigation application that comes with the system of the electronic device 210, and this embodiment of the application does not limit this.

[0083] like Figure 3 As shown, Figure 3A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device 210 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0084] It should be understood that the structures illustrated in the embodiments of the present application do not constitute a specific limitation on the electronic device 210. In other embodiments of the present application, the electronic device 210 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0085] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0086] The controller may be the nerve center and command center of the electronic device 210. The controller may generate an operation control signal based on the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0087] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly retrieve it from the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0088] The wireless communication function of the electronic device 210 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0089] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 210 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0090] Electronic device 210 implements display functionality through a GPU, display screen 194, and an application processor. A GPU is a microprocessor for image processing that connects display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. Processor 110 may include one or more GPUs that execute program instructions to generate or modify display information.

[0091] Display screen 194 is used to display images, videos, and the like. Display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-oLed, or a quantum dot light-emitting diode (QLED). In some embodiments, electronic device 210 can include one or N display screens 194, where N is a positive integer greater than one.

[0092] In an embodiment of the present application, during navigation performed by the AR navigation application of the electronic device 210 , the display screen 194 may display navigation instructions.

[0093] The electronic device 210 can implement a shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor.

[0094] The ISP processes data fed back by camera 193. For example, when taking a photo, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, which is then passed to the ISP for processing and transformed into a visible image. The ISP can also perform algorithmic optimization on image noise, brightness, and skin tone. It can also optimize parameters such as exposure and color temperature of the captured scene. In some embodiments, the ISP can be located within camera 193.

[0095] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, and then passes the electrical signal to the ISP for conversion into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the electronic device 210 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0096] In some embodiments, the camera 193 plays a crucial role in the AR navigation application of the electronic device 210 during AR navigation. The camera 193 is not only used to capture images of the real world, but also used to implement various augmented reality functions. For example, the camera 193 captures the surrounding environment of the electronic device 210 in real time to provide the user with immediate visual feedback. The AR navigation system can identify objects, surfaces, and roads around the user through the images captured by the camera 193.

[0097] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 210 by running the instructions stored in the internal memory 121. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 210 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0098] The electronic device 210 can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0099] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0100] The speaker 170A, also called a "speaker," is used to convert audio electrical signals into sound signals. The electronic device 210 can listen to music or make hands-free calls through the speaker 170A.

[0101] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device 210 receives a call or voice message, the user can place the receiver 170B close to the ear to hear the voice.

[0102] Microphone 170C, also known as a "microphone" or "speaker," converts sound signals into electrical signals. When making a call or sending a voice message, a user can place their mouth close to microphone 170C and speak, inputting the sound signal into microphone 170C. Electronic device 210 may be equipped with at least one microphone 170C.

[0103] The earphone interface 170D is used to connect wired earphones. The earphone interface 170D can be a USB interface 130, or a 3.5mm open mobile terminal platform (OMTP) standard interface, or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0104] The pressure sensor 180A is used to sense a pressure signal, and can convert the pressure signal into an electrical signal.

[0105] The gyroscope sensor 180B can be used to determine the motion posture of the electronic device 210. In some embodiments, the angular velocity of the electronic device 210 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake photography. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of shaking of the electronic device 210, and calculates the distance that the lens module needs to compensate according to the angle, so that the lens offsets the shaking of the electronic device 210 by reverse movement, thereby achieving anti-shake. The gyroscope sensor 180B can also be used for navigation and motion sensing game scenarios.

[0106] The barometric pressure sensor 180C is used to measure air pressure. In some embodiments, the electronic device 210 calculates the altitude, assists positioning and navigation by using the air pressure value measured by the barometric pressure sensor 180C.

[0107] The magnetic sensor 180D includes a Hall sensor. The electronic device 210 can detect the opening and closing of a flip cover by using the magnetic sensor 180D. In some embodiments, when the electronic device 210 is a flip phone, the electronic device 210 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. In turn, according to the detected opening and closing state of the cover or the opening and closing state of the flip cover, the electronic device 210 can set features such as automatic unlocking of the flip cover.

[0108] The acceleration sensor 180E can detect the acceleration of the electronic device 210 in various directions (generally three axes). When the electronic device 210 is stationary, the acceleration sensor 180E can detect the magnitude and direction of gravity. The acceleration sensor 180E can also be used to identify the posture of the electronic device, and can be applied to landscape / portrait switching, pedometers, and other applications.

[0109] The distance sensor 180F is used to measure distance. The electronic device 210 can measure distance by using infrared or laser. In some embodiments, in a shooting scenario, the electronic device 210 can use the distance sensor 180F to measure distance to achieve fast focusing.

[0110] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector such as a photodiode.

[0111] Ambient light sensor 180L is used to sense ambient light brightness. Electronic device 210 can adaptively adjust the brightness of display screen 194 based on the perceived ambient light brightness. Ambient light sensor 180L can also be used to automatically adjust white balance when taking photos. Ambient light sensor 180L can also work with proximity light sensor 180G to detect whether electronic device 210 is in a pocket to prevent accidental touches.

[0112] The fingerprint sensor 180H is used to collect fingerprints. The electronic device 210 can use the collected fingerprint characteristics to implement fingerprint unlocking, access application locks, fingerprint photography, fingerprint answering calls, etc.

[0113] The temperature sensor 180J is used to detect temperature.

[0114] The touch sensor 180K is also called a "touch panel." The touch sensor 180K can be disposed on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch screen." The touch sensor 180K is used to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operations can be provided via the display screen 194. In other embodiments, the touch sensor 180K can also be disposed on the surface of the electronic device 210, in a location different from that of the display screen 194.

[0115] The bone conduction sensor 180M can acquire vibration signals.

[0116] In the embodiment of the present application, the IMU may include the above-mentioned gyroscope sensor 180B, magnetic sensor 180D and acceleration sensor 180E.

[0117] The software system of the electronic device can adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system with a layered architecture is used as an example to illustrate the software structure of the electronic device.

[0118] Figure 4 A software structure diagram of the electronic device provided in an embodiment of the present application.

[0119] It is understood that a layered architecture divides software into several layers, each with distinct roles and divisions of labor. Layers communicate with each other via software interfaces. In some embodiments, the Android system may include an application layer (abbreviated as the application layer), an application framework layer (abbreviated as the framework layer), a system library, and a kernel layer.

[0120] The above application layer may include a series of application packages.

[0121] like Figure 4 As shown, the application package may include system applications. System applications refer to applications that are installed in the electronic device before leaving the factory. For example, system applications may include programs such as camera, gallery, calendar, music, short message, memo, and weather.

[0122] Application packages can also include third-party applications, which are applications that users install by downloading the installation package from an app store (or app market). Examples include map applications, food delivery applications, reading applications (such as e-books), social applications, and travel applications.

[0123] The application layer may also include an AR navigation application. The AR navigation application may be a system application or a third-party application, which is not limited here.

[0124] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0125] like Figure 4 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0126] The window manager is used to manage window programs. The window manager can obtain the display size, determine whether there is a status bar, lock the screen, take screenshots, etc.

[0127] Content providers are used to store and retrieve data and make it accessible to applications. Data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0128] The view system includes visual controls, such as those for displaying text and images. The view system is used to build applications. A display interface can consist of one or more views. For example, a display interface containing a text notification icon might include a view for displaying text and a view for displaying images.

[0129] The phone manager is used to provide communication functions for electronic devices, such as call status management (including answering, hanging up, etc.).

[0130] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0131] The notification manager enables applications to display notification information in the status bar, which can be used to convey informational messages and disappear automatically after a short stay without user interaction.

[0132] Android Runtime includes core libraries and a virtual machine. Android Runtime is responsible for scheduling and management of the Android system.

[0133] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0134] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0135] The system library can include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0136] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver and sensor driver.

[0137] The technical solutions involved in the following embodiments can all be implemented in the electronic device 210 having the above hardware structure and software architecture. In the following embodiments, the electronic device 210 is taken as a mobile phone as an example to illustrate the present solution.

[0138] In an exemplary scenario, suppose a user is using the AR navigation function on their phone to find a store in a shopping mall. The phone displays a navigation route from the phone's current location to the destination. For example, after receiving a trigger operation from the user to launch the AR navigation application, the phone launches the AR navigation application in response to the user's trigger operation and displays the navigation homepage.

[0139] Among them, the AR navigation application can be a navigation application that comes with the mobile phone system, or it can be a navigation application provided by a third-party application manufacturer, and the embodiment of this application does not limit this.

[0140] For example, Figure 5 As shown, Figure 5(a) shows a graphical user interface (GUI) of a mobile phone, which is the mobile phone desktop 510. When the mobile phone receives the user's click on the AR navigation application icon 520 on the desktop 510, the AR navigation application can be started and displayed as shown in FIG. Figure 5 The navigation homepage 530 shown in (b).

[0141] Here, the method of launching the AR navigation application by clicking the AR navigation application icon is only an example. The user can also launch the AR navigation application by other control methods, which are not limited here. For example, the user can launch the AR navigation application by voice control.

[0142] During the process of starting the AR navigation application on the mobile phone, the mobile phone can initialize the AR navigation application. For example, after the mobile phone determines that the hardware components (such as cameras, sensors, processors, etc.) have been installed and started, the mobile phone receives the user's authorization to access the camera, location information and sensor data. In response to the user's operation, the mobile phone allows the AR navigation application to access the phone's camera, location information and sensor data. The mobile phone can also calibrate sensors (such as accelerometers, gyroscope sensors, etc.) to ensure that the mobile phone collects accurate data through sensors during subsequent navigation. In addition, during the process of initializing the AR navigation on the mobile phone, the mobile phone can also load mall map data, such as downloading or updating maps of malls or garages.

[0143] It should be noted that the above-mentioned process of initializing AR navigation on a mobile phone is merely an example and is not intended to be limiting. For example, when a user is navigating in a garage or other location using the AR navigation function on a mobile phone, the mobile phone may load the map data for the garage or other location during the initialization of the AR navigation.

[0144] During the initialization of the AR navigation application on the phone, the AR navigation application can load the mall's map data and related navigation information. After the AR navigation application is initialized, the phone starts the AR navigation application, and the phone's display interface displays the mall map, such as the floor, store directory, or search box.

[0145] It's important to note that during AR navigation, the phone first performs an initialization operation to ensure the system is ready to determine the phone's location. This initialization operation is the starting point of the AR navigation process. After the initialization operation, the phone can update its location in real time during AR navigation.

[0146] When the navigation homepage 530 is displayed on the display interface of the mobile phone, after the mobile phone receives the operation of the user entering the destination (for example, store A on the 3rd floor of the shopping mall) in the search box of the navigation homepage 530, the mobile phone displays the following in response to the user's input operation: Figure 5 The navigation route planning interface 540 shown in (c) above is shown. The navigation route planning interface 540 displays the navigation route planned by the mobile phone based on the user's current location and the location of the destination. The navigation route planning interface 540 may also display an indicator indicating the user's current location, such as a user, multiple arrows indicating the navigation route, text indicating the navigation route (e.g., go straight for 10 meters and then turn right), a distance marker, and an estimated time of arrival. The navigation route is the route between the user's current location and the destination, and may include an elevator route and / or an escalator route.

[0147] The mobile phone receives the user's Figure 5 After the triggering operation of the "Start Navigation" control 550 in the navigation route planning interface 540 shown in (c) is performed, the mobile phone displays the following in response to the user's triggering operation: Figure 5 The AR navigation interface 560 shown in (d) in the figure. At this time, the AR navigation application starts to execute the navigation process. When the mobile phone controls the AR navigation application to navigate, the AR navigation interface may display navigation instructions to prompt the user to walk according to the corresponding instruction information of the navigation instructions. Figure 5 Navigation instructions can also be displayed in the AR navigation interface 560 of the mobile phone shown in (d). For example, instructions such as "go straight for 50 meters and turn right", "turn left" or "get on the elevator". The user walks according to the navigation instructions displayed in the AR navigation interface 560 and the virtual path in the video stream until the mobile phone determines that the user has reached the destination. The AR navigation application prompts the user to end the navigation process, or the AR navigation application automatically ends the navigation process. After the AR navigation application ends the navigation process, the display interface of the AR navigation application can return to the navigation homepage, or provide other service options, which are not limited here.

[0148] It should be noted that the above example of the mobile phone guiding the user by displaying navigation instructions in the AR navigation interface 560 is only an example and is not specifically limited here. For example, the mobile phone can also guide the user by playing navigation instructions by voice.

[0149] In an embodiment of the present application, during navigation by the AR navigation application of a mobile phone, the mobile phone can obtain its own location information in real time, and provide more accurate navigation instructions based on the real-time positioning of the mobile phone's location information, or adjust the navigation information based on the real-time location information. For example, the mobile phone determines that the actual route traveled by the user deviates from the planned navigation route based on the current location of the mobile phone obtained. The mobile phone can re-plan the navigation route based on the current location and display the re-planned navigation route in the AR navigation interface 560. It can be seen that during AR navigation by the mobile phone, the mobile phone can provide accurate location information based on the real-time positioning function to provide the best navigation route and navigation instructions, thereby improving the accuracy and efficiency of AR navigation.

[0150] For example, the mobile phone may use positioning technology to locate the position of the mobile phone, thereby determining the current location of the mobile phone. The positioning technology may include any one of wireless local area network (wireless fidelity, Wi-Fi) positioning technology, radio frequency identification (radio frequency identification, RFID) positioning technology, infrared positioning technology, ultrasonic positioning technology, Bluetooth positioning technology, inertial navigation positioning technology, ultra-wide band (ultra wide band, UWB) positioning technology, visible light (light emitting diode, LED) positioning technology, geomagnetic positioning technology, and visual positioning technology.

[0151] In addition, users can also interact with the AR navigation interface 560 through operations such as touching the screen, voice commands, or gestures to adjust the navigation process in real time and improve the user's interactive experience of using the AR navigation application.

[0152] In an embodiment of the present application, after the AR navigation application of the mobile phone is started, the mobile phone determines the current location of the mobile phone through the positioning function. In response to the operation of inputting the destination, the mobile phone displays the navigation route planning interface, and in response to the operation of starting navigation, it displays the AR navigation interface. Among them, the AR navigation interface includes navigation instructions. Here, the navigation instructions are determined based on the first image captured by the camera and the current location of the mobile phone. During the navigation process of the AR navigation application of the mobile phone, when the mobile phone determines that the current location of the mobile phone is on the escalator or in the elevator based on the second image captured by the camera, the mobile phone stops the positioning function and displays the navigation instructions on the AR navigation interface based on the navigation route. In this way, the problem of positioning failure or large drift of positioning on the escalator or elevator is avoided, which causes the AR navigation application to re-plan the wrong navigation route based on the wrong positioning information.

[0153] It should be explained that the mobile phone can combine the first image captured by the camera with the IMU data to determine the current position and orientation of the mobile phone. The mobile phone can also overlay virtual information in the user's field of view based on the first image captured by the camera to provide the user with intuitive navigation guidance, such as navigation arrows, road signs, and point of interest markers. This allows the mobile phone to dynamically update the navigation guidance based on the real-time image captured by the camera.

[0154] In some embodiments, when the phone determines that it is currently on an escalator or in an elevator, the phone stops initializing operations related to the positioning function. Specifically, the phone does not perform any operations related to the positioning function. This prevents initialization from failing due to inaccurate data (e.g., sensor-related data) detected during initialization in dynamic environments such as escalators or elevators.

[0155] It should be noted that during navigation using a mobile phone's AR navigation app, even when the phone is on an escalator or in an elevator and the phone's positioning function or initialization process has stopped, the phone still controls the AR navigation app to navigate according to the planned navigation route. Therefore, even though the phone's positioning function has stopped, it still navigates according to the planned navigation route, which does not affect the navigation process and does not cause issues such as positioning failure or positioning drift that reduce navigation accuracy.

[0156] The following is an example of a user using a mobile phone to find a store in a shopping mall, and the navigation route planned by the mobile phone's AR navigation application includes an escalator route, to exemplify the navigation method provided in the embodiment of the present application. Figure 6 A flowchart of a navigation method provided in an embodiment of the present application is shown as follows: Figure 6 As shown, the navigation method may include the following processes.

[0157] Step 601: After the mobile phone controls the AR navigation application to complete the initialization operation, the AR navigation application is started.

[0158] The process of the mobile phone controlling the initialization operation of the AR navigation application can be found in the above introduction and will not be repeated here.

[0159] In some embodiments, when the mobile phone starts the AR navigation application in response to the user's operation, the AR navigation application can automatically turn on the positioning function so that the mobile phone can obtain the current location information of the mobile phone in real time during the AR navigation process.

[0160] Of course, the mobile phone can also open the positioning function in other ways, and the embodiment of the present application does not limit the way of opening the positioning function. For example, after the mobile phone receives the operation of opening the positioning function triggered by the user, it responds to the operation and opens the positioning function.

[0161] During AR navigation on a mobile phone, turning on the positioning function can not only determine the current location of the mobile phone in real time to provide more accurate navigation paths and instructions, but also update the location information of the mobile phone in real time to adjust the navigation route and instructions during AR navigation to ensure that the user follows the correct route.

[0162] Step 602: The mobile phone displays a navigation route in response to the user inputting a destination.

[0163] Among them, the navigation route includes escalator route.

[0164] After the mobile phone starts the AR navigation application, after the mobile phone receives the user's input of the destination, the navigation route is displayed on the display interface of the mobile phone in response to the user's input operation.

[0165] Step 603: In response to the navigation start operation, the mobile phone starts AR navigation based on the navigation route.

[0166] For example, the mobile phone can be based on Figure 5 Start navigating along the route shown in (c).

[0167] Step 604: The mobile phone obtains images and IMU data collected by the camera.

[0168] Among them, IMU data can include the acceleration, angular velocity and magnetic field of the mobile phone.

[0169] For example, assuming that the IMU in a mobile phone includes an acceleration sensor, an angular velocity sensor, and a magnetic sensor, the IMU data obtained by the mobile phone includes the acceleration, angular velocity, and magnetic field during the movement of the mobile phone.

[0170] In an embodiment of the present application, the mobile phone can obtain images captured by the camera in real time or periodically. For example, the mobile phone can obtain images captured by the camera every 0.5 seconds or 1 second.

[0171] Step 605: The mobile phone determines the camera pose based on the image collected by the camera and the IMU data.

[0172] The camera pose refers to the position and orientation of the camera in three-dimensional space. The camera pose usually consists of a rotation matrix and a translation vector. For example, the camera pose is expressed as (r, x, y, z), where r is the rotation matrix and (x, y, z) is the translation vector. Typically, the camera pose can be expressed as a 4×4 transformation matrix, namely the camera pose matrix, which contains information such as the camera's position and orientation. The first 3×3 matrices of the 4×4 transformation matrix are the rotation matrices, and the last column is the translation vector. The rotation matrix is ​​used to describe the rotation of the camera relative to the world coordinate system (or a reference coordinate system). The rotation matrix r can represent the rotation of the camera around three axes (usually the X-axis, Y-axis, and Z-axis). The translation vector is used to describe the position of the camera in the world coordinate system. The translation vectors (x, y, z) represent the position of the camera on the X-axis, Y-axis, and Z-axis, respectively.

[0173] In an embodiment of the present application, during navigation by the mobile phone's AR navigation application, the mobile phone can obtain images and IMU data captured by the camera in real time to determine the camera pose based on the images and IMU data captured by the camera.

[0174] In some embodiments, after the mobile phone obtains the image captured by the camera, it uses a computer vision algorithm (for example, the scale-invariant feature transform (SIFT) algorithm) to perform feature extraction on the image captured by the camera to extract image feature points. For example, feature points can be corners, edges, color changes, etc. The mobile phone matches the feature points of the image currently captured by the camera with the feature points of the previous frame of image captured by the camera to obtain a feature matching result. Then, the mobile phone first fuses the IMU data with the feature matching results of the image, and then uses an optimization algorithm to determine the position and orientation of the camera in three-dimensional space. For example, the mobile phone can use nonlinear optimization or direct linear transformation to determine the position and orientation of the camera in three-dimensional space.

[0175] When the mobile phone performs data fusion on the feature matching results of the IMU data and the image, the following data fusion methods can be used: extended Kalman filter (EKF), unscented Kalman filter (UKF) or feature-based simultaneous localization and mapping (SLAM).

[0176] It should be noted that the above-mentioned process of determining the camera pose based on the images captured by the camera and the IMU data is only an example. The phone can also use other camera pose estimation methods (such as direct methods, etc.) to determine the camera pose, which is not limited here. Among them, the direct method is a camera pose estimation method that does not rely on feature points. It determines the camera pose by directly comparing the pixel value differences between the two images captured by the camera.

[0177] The phone determines the camera's position here to subsequently determine whether the phone has entered the escalator area. In fact, during AR navigation on a phone, camera position is crucial to achieving the augmented reality experience. For example, as the phone's position changes, the camera's position is constantly updated. By determining the camera's position in real time, the phone can accurately overlay virtual information (such as navigation arrows, road signs, and tooltips) onto the user's real-world view, ensuring that the virtual information is correctly positioned within the user's field of view.

[0178] Step 606: The mobile phone obtains the pre-generated bounding box.

[0179] Among them, the bounding box is an algorithm for solving the optimal bounding space of a discrete point set. The basic idea is to use a geometric body with a slightly larger volume and simpler characteristics (called a bounding box) to approximate a complex geometric object. For example, a bounding box can be a rectangular area used to represent the boundary of an object in an image. Of course, the bounding box can also be any other shape, for example, a square area, a circular area, a diamond area, etc. The shape of the bounding box is not limited in the embodiments of the present application.

[0180] In AR navigation, bounding boxes can be used to identify and locate specific landmarks or points of interest in images to provide navigation information in an augmented reality environment.

[0181] It should be noted that the bounding box is pre-generated by the phone based on the offline map. During this AR navigation process, the phone can directly obtain the bounding box, without having to generate it again during the AR navigation process.

[0182] In some embodiments, after the mobile phone obtains the offline map from the server or the storage space of the device, it can first pre-process the offline map, such as adjusting the brightness, contrast and color. Then, the mobile phone performs feature extraction on the offline map to extract features from the map image, such as corners, edges, color blocks, etc. Here, the mobile phone performs feature extraction on the offline map to better identify and locate specific objects in the image. Furthermore, the mobile phone can use a target detection algorithm to perform target detection on the map image to detect the target object and determine the position and size of each target object. For each target object detected, the mobile phone generates a bounding box, which contains the position and size of the target object. The center of the bounding box is located at the center of mass of the target object and covers the entire target object.

[0183] The above-mentioned target detection algorithm can be YOLO, single shot multibox detector (SSD), faster region-based convolutional neural network (Faster R-CNN), deep full convolution single stage target detection (RetinaNet), etc. The actual target detection algorithm used in the embodiment of the present application is not specifically limited.

[0184] For example, when the bounding box generated by the mobile phone based on the offline map is an escalator bounding box, the escalator bounding box may include the escalator's boundaries (e.g., the outline of the handrails and steps), dimensions (escalator width and height), location information (i.e., the escalator's specific location on the map, including its horizontal and vertical positions), orientation, type (e.g., straight escalator or hand escalator), and status information (e.g., running or stationary). The escalator bounding box is used to indicate the escalator's location and size in the offline map.

[0185] In step 607 , the mobile phone determines whether the current position of the mobile phone is within the escalator area (ie, the target area) based on the bounding box and the camera posture.

[0186] The escalator area refers to the physical space where the escalator is located, including the escalator itself and the space around the escalator, and is used to represent the spatial range of the escalator in the real world.

[0187] Since the bounding box coordinates belong to the world coordinate system, while the camera coordinates belong to the camera coordinate system, the bounding box and the camera do not belong to the same coordinate system. Therefore, the mobile phone converts the coordinates of the escalator bounding box generated according to the offline map into the camera coordinate system. Based on the coordinates of the bounding box and the camera position in the same coordinate system, the mobile phone can determine whether the current position is within the escalator area.

[0188] Specifically, the phone first performs intrinsic and extrinsic calibration on the camera. This involves determining the camera's intrinsic parameter matrix (e.g., focal length, principal point, etc.) and extrinsic parameter matrix (i.e., rotation matrix and translation matrix). The phone then uses the camera's extrinsic matrix to transform the coordinates of the escalator's bounding box in the world coordinate system to the camera coordinate system.

[0189] For example, Figure 7 As shown, the mobile phone determines the vertex coordinates corresponding to the escalator bounding box abcd in the escalator area in the world coordinate system as: vertex a (x1, y1, z1), vertex b (x2, y1, z1), vertex c (x1, y2, z1), vertex d (x2, y2, z1), where x, y, and z correspond to the X, Y, and Z axis components of the coordinate point, respectively. The mobile phone can then use the camera's extrinsic parameter matrix to transform the coordinates in the world coordinate system into the camera coordinate system. For example, the mobile phone uses the following formula (1) to transform a point in the world coordinate system into the camera coordinate system.

[0190] P_camera=R*P_world+T formula (1).

[0191] Among them, P_camera is the coordinate value corresponding to the point in the camera coordinate system, P_world is the coordinate value corresponding to the point in the world coordinate system, R is the rotation matrix, and T is the translation matrix.

[0192] The mobile phone can use the above formula (1) to convert the coordinates of each vertex of the escalator bounding box in the world coordinate system of the escalator area into the camera coordinate system to obtain the coordinates of each vertex of the escalator bounding box in the camera coordinate system.

[0193] In some embodiments, the transformed escalator bounding box may be distorted due to the perspective effect of the camera lens. In this case, the phone can adjust the transformed escalator bounding box to adapt to the camera's perspective effect and ensure that the boundaries of the escalator bounding box correctly represent the escalator area in the camera coordinate system.

[0194] In an embodiment of the present application, after the mobile phone converts the coordinates of the escalator bounding box to the camera coordinate system, the mobile phone can determine whether the current position of the mobile phone is within the escalator area based on the coordinate values ​​of the escalator bounding box and the coordinate values ​​of the camera posture in the same coordinate system.

[0195] As a possible implementation method, when the escalator bounding box is a rectangular area or a square area, after the mobile phone determines the camera pose, the mobile phone can compare the camera pose with the coordinate values ​​of each vertex of the escalator bounding box in the camera coordinate system to determine whether the current position of the mobile phone is within the escalator area.

[0196] For example, the mobile phone can use the following formula (2) to compare the camera pose with the coordinate values ​​of each vertex of the escalator bounding box in the camera coordinate system to determine whether the current position of the mobile phone is within the escalator area.

[0197]

[0198] Among them, flag =1, indicating that the current location of the mobile phone is within the escalator area, in flag =0, indicating that the current location of the mobile phone is outside the escalator area, x max is the maximum value of the horizontal coordinate value of each vertex of the escalator bounding box, y max is the maximum value of the ordinate value of each vertex of the escalator bounding box, x is the abscissa value of the camera pose of the mobile phone, and y is the ordinate value of the camera pose of the mobile phone.

[0199] like Figure 7 As shown in (a), when the mobile phone is at position A1, the mobile phone obtains the camera pose (r A , x A ,y A , z A ), the phone will be at the x position of the camera at position A1 A and y A The value of is compared with the vertex coordinates of the escalator bounding box. The mobile phone determines that x2>x A >x1,y A >y2, the mobile phone determines that the current position A is outside the escalator area.

[0200] Continue as Figure 7 As shown in (a), when the mobile phone is at position B1, the mobile phone obtains the camera pose (r B , x B ,y B , z B ), the phone will be at the x position of the camera pose at position B1 B and y B The value of is compared with the vertex coordinates of the escalator bounding box. The mobile phone determines that x2>x B >x1,y2>y B >y1, the mobile phone determines that the current position B is within the escalator area.

[0201] As another possible implementation, when the escalator bounding box is a rectangular or square area, after the mobile phone determines the camera pose, it can calculate the difference between the horizontal coordinate value of the camera pose and the horizontal coordinate value of the center point of the escalator bounding box to obtain a first difference, and calculate the difference between the vertical coordinate value of the camera pose and the vertical coordinate value of the center point of the escalator bounding box to obtain a second difference. The mobile phone determines whether the current position of the mobile phone is within the escalator area based on the relationship between the absolute value of the first difference and the first difference threshold, and the relationship between the absolute value of the second difference and the second difference threshold.

[0202] The first difference threshold may be half of the length of the escalator bounding box, and the second difference threshold may be half of the width of the escalator bounding box.

[0203] In one case, when the mobile phone determines that the absolute value of the first difference is less than the first difference threshold and the absolute value of the second difference is less than the second difference threshold, the mobile phone determines that the current location is within the escalator area.

[0204] For example, Figure 7 As shown in (b), when the mobile phone is at position B2, the mobile phone obtains the camera pose r at position B2 B , x B ,y B , z B ), the phone calculates the x position of the camera at position B2 B The first difference between the value of and the horizontal coordinate x0 of the center point H of the escalator bounding box, and x B and a second difference from the ordinate y0 of the center point H. The mobile phone determines that the first difference is less than the first difference threshold, and the second difference is also less than the second difference threshold, and the mobile phone determines that the current position is within the escalator area.

[0205] In another case, when the mobile phone determines that the absolute value of the first difference is greater than the first difference threshold and the absolute value of the second difference is less than the second difference threshold, the mobile phone determines that the current position is not within the escalator area.

[0206] For example, Figure 7 As shown in (b), when the mobile phone is at position C2, the mobile phone obtains the camera pose (r C , x C ,y C , z C ), the phone calculates the x position of the camera at position C2 C The first difference between the value of , and the horizontal coordinate x0 of the center point H of the escalator bounding box, and y C and a second difference from the ordinate y0 of the center point H. The mobile phone determines that the absolute value of the first difference is greater than the first difference threshold, and the absolute value of the second difference is less than the second difference threshold, and the mobile phone determines that the current position is outside the escalator area.

[0207] In another case, when the mobile phone determines that the absolute value of the first difference is less than the first difference threshold and the absolute value of the second difference is greater than the second difference threshold, the mobile phone determines that the current location is not within the escalator area.

[0208] For example, Figure 7 As shown in (b), when the mobile phone is at position A2, the mobile phone obtains the camera pose (r A , x A ,y A , z A ), the phone calculates the x position of the camera at position A2 A The first difference between the value of , and the horizontal coordinate x0 of the center point H of the escalator bounding box, and y A and a second difference from the ordinate y0 of the center point H. The mobile phone determines that the absolute value of the first difference is less than the first difference threshold, and the absolute value of the second difference is greater than the second difference threshold, and the mobile phone determines that the current position is outside the escalator area.

[0209] In another case, when the mobile phone determines that the absolute value of the first difference is greater than the first difference threshold and the absolute value of the second difference is greater than the second difference threshold, the mobile phone determines that the current location is not within the escalator area.

[0210] For example, Figure 7 As shown in (b), when the mobile phone is at position D2, the mobile phone obtains the camera pose (r D , x D ,y D , z D ), the phone calculates the x position of the camera at position D2 A The first difference between the value of , and the horizontal coordinate x0 of the center point H of the escalator bounding box, and y D and a second difference from the ordinate y0 of the center point H. The mobile phone determines that the absolute value of the first difference is greater than the first difference threshold, and the absolute value of the second difference is greater than the second difference threshold, and the mobile phone determines that the current position is outside the escalator area.

[0211] As another possible implementation, when the escalator bounding box is a circular area, after the mobile phone determines the camera pose, it can calculate the difference between the horizontal coordinate value of the camera pose and the horizontal coordinate value of the center point of the escalator bounding box to obtain a third difference, and calculate the difference between the vertical coordinate value of the camera pose and the vertical coordinate value of the center point of the escalator bounding box to obtain a fourth difference. The mobile phone determines whether the current position of the mobile phone is within the escalator area based on the relationship between the absolute value of the third difference and the third difference threshold, as well as the relationship between the absolute value of the second difference and the third difference threshold. The third difference threshold is the radius of the circular area.

[0212] In one case, when the mobile phone determines that the absolute value of the third difference is less than the third difference threshold and the absolute value of the fourth difference is also less than the third difference threshold, the mobile phone determines that the current location is within the escalator area.

[0213] For example, Figure 7 As shown in (c), when the mobile phone is at position B3, the mobile phone obtains the camera pose r at position B3 B , x B ,y B , z B ), the phone calculates the x position of the camera at position B3 B The third difference between the value of , and the horizontal coordinate x0 of the center point O of the escalator bounding box, and x B and a fourth difference from the ordinate y0 of the center point O. The mobile phone determines that the absolute value of the third difference and the absolute value of the fourth difference are both less than the third difference threshold, and the mobile phone determines that the current position is within the escalator area.

[0214] In another case, when the mobile phone determines that the absolute value of the third difference is greater than the third difference threshold and the absolute value of the fourth difference is less than the third difference threshold, the mobile phone determines that the current location is not within the escalator area.

[0215] For example, Figure 7 As shown in (c), when the mobile phone is at position C3, the mobile phone obtains the camera pose (r C , x C ,y C , z C ), the phone calculates the x position of the camera at position C3 C The third difference between the value of , and the horizontal coordinate x0 of the center point O of the escalator bounding box, and y C and a fourth difference from the ordinate y0 of the center point O. The mobile phone determines that the absolute value of the first difference is greater than the first difference threshold, and the absolute value of the second difference is less than the second difference threshold, and the mobile phone determines that the current position is outside the escalator area.

[0216] In another case, when the mobile phone determines that the absolute value of the third difference is less than the third difference threshold and the absolute value of the fourth difference is greater than the third difference threshold, the mobile phone determines that the current location is not within the escalator area.

[0217] For example, Figure 7 As shown in (c), when the mobile phone is at position A3, the mobile phone obtains the camera pose (r A , x A ,y A , z A ), the phone calculates the x position of the camera at position A3 Aa third difference value of the value of the x-coordinate of the center point O of the escalator bounding box and the x-coordinate of the center point O of the mobile phone A a fourth difference value of the value of the y-coordinate of the center point O of the escalator bounding box and the y-coordinate of the center point O of the mobile phone When the absolute value of the third difference value is less than a third difference threshold value and the absolute value of the fourth difference value is greater than a second difference threshold value, the mobile phone determines that the current position is outside the escalator region.

[0218] It should be explained that the method for determining whether the mobile phone enters the escalator region according to the camera pose and the coordinate value of the escalator bounding box is only described as an example, and the embodiments of the present application are not limited thereto.

[0219] In the embodiments of the present application, when the mobile phone determines that the current position of the mobile phone is located in the escalator region, the mobile phone continues to perform step 608, otherwise, step 612 is performed.

[0220] It can be understood that the escalator region also includes the space around the escalator, so when the mobile phone determines that the current position of the mobile phone is located in the escalator region, the mobile phone may be located around the escalator and not on the escalator. In this case, the mobile phone needs to continue to determine whether the current position of the mobile phone is located on the escalator. The specific determination process is described below in the process of step 608.

[0221] Step 608, if the mobile phone determines that the current position of the mobile phone is located in the escalator region, the mobile phone performs an escalator detection process.

[0222] The escalator detection process refers to a process of determining whether the current position is located on the escalator.

[0223] In some embodiments, during the process of controlling the AR navigation to navigate, the mobile phone can acquire the image captured by the camera in real time, perform target detection on the image captured by the camera, and determine whether the current position of the mobile phone is located on the escalator according to the detection result.

[0224] Optionally, as shown in Figure 8 After the mobile phone acquires the image captured by the camera, the mobile phone can perform preprocessing (such as scaling, rotating, denoising, enhancing, etc.) on the image captured by the camera to obtain a preprocessed image. Then, the mobile phone inputs the preprocessed image into the trained target detection network to determine whether the current position of the mobile phone is located on the escalator according to the detection result output by the target detection network.

[0225] In some embodiments, the process of training the target detection network by a mobile phone is as follows: First, the mobile phone obtains an image dataset containing an escalator as a training sample. The image dataset includes images of escalators captured at different angles, lighting conditions, or backgrounds. Then, the mobile phone annotates the images in the image dataset, annotating the escalator in each image with labels such as a bounding box and category. The mobile phone can also perform preprocessing operations such as scaling, rotating, cropping, and flipping on the images in the image dataset to obtain a preprocessed image dataset. The mobile phone can use the target detection algorithm to build a network model and configure the network parameters. The mobile phone inputs the preprocessed image dataset into the network model for training, and obtains a trained target detection network by adjusting the network parameters.

[0226] Similarly, the above-mentioned target detection algorithm can also be YOLO, SSD, Faster R-CNN, RetinaNet, etc. The embodiment of this application does not specifically limit the target detection algorithm actually used.

[0227] For example, Figure 9 This is a network model structure diagram for target detection provided in an embodiment of the present application. Figure 9 As shown in the figure, after the phone inputs the original image into the feature pyramid network (FPN) on the far left, the FPN layer outputs feature maps of different sizes. This means that the FPN extracts features at different levels from the original image. P5, P4, and P3 represent different feature layers with different resolutions. H×W×{1024, 512, 256} represents the number of channels in each feature layer, which are 1024, 512, and 256, respectively. This means that in the network model, each feature layer has two branches: one for classification and the other for regression. H×W×256 is the dimension of the feature map, where H and W represent the height and width of the image, and 256 represents the number of channels. The three main components of the network model are the classifier, the regression head, and the intersection over union (IoU) model, which are responsible for classification, regression, and calculating IoU, respectively. H×W×C is the output dimension of the classifier, where H and W represent the height and width of the image, and C represents the number of categories. H×W×4 is the output dimension of the regression head, where H and W represent the height and width of the image, and 4 represents the four coordinate values ​​corresponding to each pixel.

[0228] Step 609: The mobile phone determines whether the current location of the mobile phone is on the escalator.

[0229] In an embodiment of the present application, the mobile phone can determine whether the current location of the mobile phone is on the escalator based on the detection results output by the target detection network.

[0230] In one case, if the mobile phone determines that the detection result output by the object detection network includes an escalator, the mobile phone determines that the current location is on the escalator. If the mobile phone determines that the detection result output by the object detection network does not include an escalator, the mobile phone determines that the current location is not on the escalator.

[0231] It should be noted that, when the mobile phone determines that the detection result includes the steps and / or handrails of the escalator, it is determined that the current position of the mobile phone is on the escalator.

[0232] In step 610, the mobile phone determines that the current location is on the escalator, stops the positioning function, and performs AR navigation according to the navigation route.

[0233] In the embodiment of the present application, the mobile phone determines that its current location is on the escalator based on the detection result of the target detection network, and the mobile phone stops the positioning function, that is, the mobile phone stops obtaining the current location of the mobile phone.

[0234] It can be understood that due to the motion of the escalator, the mobile phone may experience positioning failure or inaccurate positioning. If the mobile phone continues to perform real-time positioning while on the escalator, there is a possibility that the AR navigation application will provide incorrect route guidance instructions based on the positioning results, or switch to another navigation route. In the embodiment of the present application, when the mobile phone determines that the current location of the mobile phone is on the escalator, the mobile phone stops the positioning function, thus avoiding the problem of the AR navigation application switching to the wrong navigation route.

[0235] In some embodiments, when the mobile phone determines that the current location of the mobile phone is on an escalator based on the detection results of the target detection network, the mobile phone can display a prompt message on the AR navigation interface of the AR navigation application to prompt the user to confirm that the current location is on the escalator. When the mobile phone detects the user's operation to confirm that the current location is on the escalator in the AR navigation interface, in response to the user's operation, the mobile phone can determine that the current location is on the escalator. It can be seen that by interacting with the user on the AR navigation interface, the mobile phone can more accurately determine that the current location of the mobile phone is on the escalator.

[0236] For example, Figure 10 As shown, when the phone determines that its current location is on an escalator based on the detection results of the target detection network, the phone can display a prompt message 1020, namely, "Is the current location on an escalator?", on the AR navigation interface 1010 of the AR navigation application. After the phone receives the user's trigger operation on the control "Yes", in response to the user's trigger operation, the phone can determine that the current location is on the escalator and suspend the positioning function.

[0237] In some embodiments, the mobile phone can also prompt the user to determine whether the current position is on the escalator by voice interaction with the user. For example, when the mobile phone determines that the current position is on the escalator, the mobile phone plays a voice message "Have you been on the escalator now?" by voice play. After the mobile phone receives the voice message replied by the user, which includes an affirmative tone, the mobile phone determines that the current position of the mobile phone is indeed on the escalator, and the mobile phone stops the positioning function.

[0238] In one case, when the mobile phone determines that the current position of the mobile phone is on the escalator, the mobile phone can control the positioning function of the AR navigation application to be closed.

[0239] In another case, when the mobile phone determines that the current position of the mobile phone is on the escalator, the mobile phone can control the positioning function of the mobile phone itself to be closed. For example, the mobile phone can control the switch of the GPS function of the mobile phone to be in the off state.

[0240] It should be explained that the above method of suspending the real-time positioning function of the mobile phone is only an example, and the mobile phone can also suspend the positioning function of the AR navigation application in other ways, which are not limited here.

[0241] In the embodiments of the present application, when the mobile phone stops the positioning function, the AR navigation interface of the mobile phone can display prompt information to prompt that the positioning function is in the closed state, so as to avoid the problem that the user actively updates the navigation route when the current position of the mobile phone is on the escalator, and cannot accurately obtain the current position. For example, as shown in Figure 11 When the mobile phone stops the positioning function, the mobile phone can display prompt information 1120, i.e. "the current position is on the escalator", in the AR navigation interface 1110 of the AR navigation application.

[0242] In addition, the mobile phone can also prompt the user that the current position is on the escalator by playing voice information. For example, when the mobile phone determines that the current position is on the escalator, the mobile phone plays voice information "the current position is on the escalator" by voice play.

[0243] In addition, during the process of suspending the real-time positioning function of the mobile phone, the mobile phone no longer determines the camera pose according to the image currently collected by the camera, but the AR navigation application of the mobile phone still navigates according to the pre-planned navigation route. Thus, the problem that the navigation application changes the navigation route according to the wrong positioning information when the navigation route is the escalator route is avoided, and the navigation accuracy is reduced.

[0244] In some embodiments, when the mobile phone determines that the current location of the mobile phone is on an escalator, the mobile phone stops initialization operations related to the positioning function. That is, the mobile phone does not perform any operations related to the positioning function. This avoids the problem of inaccurate detection data (e.g., sensor-related data) when performing initialization operations in dynamic environments such as escalators or elevators, resulting in unsuccessful initialization operations.

[0245] Step 611: When the mobile phone determines that the current location is not on the escalator, the mobile phone turns on the positioning function.

[0246] In this embodiment of the present application, after the phone pauses its positioning function on the escalator, it continues to acquire images captured by the camera and performs real-time detection on these images using the target detection network to determine whether the phone is still on the escalator based on the detection results. When the phone determines that it has left the escalator, it activates the positioning function, thereby improving the accuracy of subsequent AR navigation.

[0247] In some embodiments, after the mobile phone obtains the third image captured by the camera, when it is determined that the current location of the mobile phone is not on the escalator based on the detection result of the target detection network on the third image, the mobile phone turns on the positioning function again, and after using the positioning function to determine the current location of the mobile phone, displays navigation instructions on the AR navigation interface.

[0248] In one scenario, when the phone determines that it has left the escalator, it can display a prompt to remind the user to confirm that the phone has left the escalator. If the phone detects that the user has confirmed in the AR navigation interface that they are not currently on the escalator, the phone can determine that they have left the escalator in response to the user's operation. This shows that by interacting with the user in the AR navigation interface, the phone can more accurately determine whether the phone is on the escalator.

[0249] For example, Figure 12 As shown, when the phone determines that the user has left the escalator based on the detection results of the target detection network, the phone can display a prompt message 1220, namely, "Have you left the escalator?", on the AR navigation interface 1210 of the AR navigation application. After the phone receives the user's trigger operation on the control "Yes", in response to the user's trigger operation, the phone can determine that the user has left the escalator at the current moment and activate the positioning function. If the phone detects the user's trigger operation on the control "No", in response to the user's trigger operation, the phone continues to suspend the positioning function.

[0250] In another scenario, the phone can also prompt the user to confirm the current time of leaving the escalator through voice interaction. For example, when the phone determines that the current time has left the escalator, the phone plays a voice message "Excuse me, have you left the escalator now?" After receiving the user's affirmative voice message reply, the phone confirms that the phone has left the escalator at the current time and activates the positioning function.

[0251] It should be noted that the above-mentioned mobile phone reminds the user that the mobile phone is not on the escalator at the current moment by displaying a prompt message or playing a voice message, which is only an example. The mobile phone can also remind the user in other ways, for example, the mobile phone can play a voice message while displaying the prompt message.

[0252] In step 612, the mobile phone executes the AR navigation process normally.

[0253] In an embodiment of the present application, the mobile phone controls the AR navigation application to perform navigation, that is, displays navigation instructions on the AR navigation interface until the mobile phone determines that the current location of the mobile phone is at the destination, at which point the mobile phone controls the AR navigation application to stop navigation. The navigation instructions are determined based on the image captured by the camera and the current location of the mobile phone.

[0254] To sum up, in the embodiment of the present application, since the navigation route planned by the mobile phone according to the destination includes the escalator route, in order to avoid the problem of inaccurate positioning when the mobile phone is on the escalator, which causes the AR navigation application to update to the wrong navigation route, during the process of the mobile phone controlling the AR navigation application for navigation, when the mobile phone determines that the current position is on the escalator, the mobile phone suspends the positioning function, and the mobile phone continues to navigate based on the pre-planned navigation route until the mobile phone determines that the current position is not on the escalator, that is, it determines that the user has left the escalator, and the mobile phone turns on the positioning function again, thereby improving the accuracy of AR navigation and improving the user's travel experience.

[0255] It should be noted that the above embodiment is illustrated by taking the AR navigation application set in the mobile phone to execute the navigation process as an example. Of course, other electronic devices can also execute the above navigation process, such as smart watches, smart bracelets, tablets or smart glasses, etc. The specific implementation process can be referred to in the above embodiment and will not be described in detail here.

[0256] In addition, the above embodiment is only illustrated by taking the navigation route including the escalator route as an example. When the navigation route includes the elevator route, the method of the above embodiment is also applicable.

[0257] For example, in a scenario where the navigation route planned by the mobile phone based on the destination input by the user includes an elevator route, when the mobile phone detects that the current location of the mobile phone is inside the elevator, the mobile phone suspends the real-time positioning function until the mobile phone detects that it has left the elevator at the current moment, and then turns on the real-time positioning function again, avoiding the problem of inaccurate positioning of the mobile phone when it is in the elevator, which affects the navigation accuracy.

[0258] As another example, in a scenario where the navigation route planned by the mobile phone based on the destination input by the user includes both escalator routes and elevator routes, when the mobile phone detects that the current location of the mobile phone is on the escalator or in the elevator, the mobile phone suspends the real-time positioning function until the mobile phone detects that it has left the escalator or elevator at the current moment, and then turns on the real-time positioning function again, avoiding the problem of inaccurate positioning when the mobile phone is on the escalator or in the elevator, affecting the navigation accuracy.

[0259] like Figure 13 As shown, an embodiment of the present application discloses an electronic device, which may be the above-mentioned mobile phone. The electronic device may specifically include: a touch screen 1301, wherein the touch screen 1301 includes a touch sensor 1306 and a display screen 1307; one or more processors 1302; a memory 1303; one or more application programs (not shown); and one or more computer programs 1304. The above-mentioned components may be connected via one or more communication buses 1305. The one or more computer programs 1304 are stored in the above-mentioned memory 1303 and are configured to be executed by the one or more processors 1302. The one or more computer programs 1304 include instructions that can be used to perform the relevant steps in the above-mentioned embodiment.

[0260] It is understandable that, in order to realize the above functions, the above-mentioned electronic devices and the like include hardware structures and / or software modules corresponding to the execution of each function. Those skilled in the art should easily realize that, in combination with the units and algorithm steps of each example described in the embodiments disclosed herein, the embodiments of the present application can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the embodiments of the present invention.

[0261] The embodiment of the present application can divide the functional modules of the above-mentioned electronic device etc. according to the above-mentioned method example. For example, each functional module can be divided corresponding to each function, or two or more functions can be integrated into one processing module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of software functional modules. It should be noted that the division of modules in the embodiment of the present invention is schematic and is only a logical function division. There may be other division methods in actual implementation.

[0262] A schematic diagram of a possible configuration of the electronic device involved in the above embodiment, in which each functional module is divided according to its function, may include a display unit, a transmission unit, and a processing unit. It should be noted that all relevant content of each step involved in the above method embodiment can be referred to in the functional description of the corresponding functional module and will not be repeated here.

[0263] An embodiment of the present application further provides an electronic device comprising one or more processors and one or more memories. The one or more memories are coupled to the one or more processors, and the one or more memories are used to store computer program code. The computer program code comprises computer instructions. When the one or more processors execute the computer instructions, the electronic device executes the above-mentioned related method steps to implement the AR navigation method in the above-mentioned embodiment.

[0264] An embodiment of the present application also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the AR navigation method in the above-mentioned embodiment.

[0265] An embodiment of the present application further provides a computer program product, which includes computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the above-mentioned related method steps to implement the AR navigation method in the above-mentioned embodiment.

[0266] In addition, an embodiment of the present application also provides a device, which can specifically be a chip, component or module, and the device may include a connected processor and memory; wherein the memory is used to store computer-executable instructions, and when the device is running, the processor can execute the computer-executable instructions stored in the memory to enable the device to execute the AR navigation method performed by the electronic device in the above-mentioned method embodiments.

[0267] Among them, the electronic device, computer-readable storage medium, computer program product or device provided in this embodiment are all used to execute the corresponding methods provided above. Therefore, the beneficial effects that can be achieved can refer to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0268] Through the description of the above embodiments, those skilled in the art will clearly understand that for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. The specific working processes of the above-described systems, devices, and units can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0269] The functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0270] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as flash memory, mobile hard disk, read-only memory, random access memory, magnetic disk or optical disk.

[0271] The above is only a specific embodiment of the present application, but the scope of protection of this application is not limited to this. Any changes or substitutions within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. An augmented reality (AR) navigation method, characterized in that: The method comprises: After launching the AR navigation app, enable the positioning function to determine the current location of the electronic device; In response to an operation of inputting a destination, displaying a navigation route planning interface, the navigation route planning interface including a navigation route determined based on a current location of the electronic device and the destination; In response to the operation of starting navigation, displaying an AR navigation interface, the AR navigation interface including a navigation guide, the navigation guide being determined based on the first image captured by the camera and the current position of the electronic device; When it is determined that the current position of the electronic device is at a preset position based on the second image captured by the camera, the positioning function is stopped, and navigation instructions are displayed on the AR navigation interface based on the navigation route; the preset position includes an escalator.

2. The method according to claim 1, characterized in that The method further comprises: Display a first prompt message, where the first prompt message is used to prompt that the user is currently at a preset location, and continuously display navigation instructions on the AR navigation interface.

3. The method according to claim 1, characterized in that After stopping the positioning function, the method further includes: Acquire a third image captured by the camera; Determining that the current position of the electronic device is not at the preset position based on the third image, turning on the positioning function, and displaying navigation guidance on the AR navigation interface after using the positioning function to determine the current position of the electronic device.

4. The method according to claim 3, characterized in that The method further comprises: A second prompt message is displayed, where the second prompt message is used to prompt that the user has left the preset location.

5. The method according to any one of claims 1 to 4, characterized in that Before determining that the current position of the electronic device is at a preset position, the method further includes: It is determined that the current position of the electronic device is in a target area, where the target area is an area corresponding to the escalator.

6. The method according to claim 5, characterized in that Determining that the current location of the electronic device is in the target area includes: Get inertial measurement unit IMU data; Determining a camera pose based on the second image and the IMU data; When the current position of the electronic device is determined to enter the escalator bounding box based on the camera pose, it is determined that the current position of the electronic device is in the target area. The escalator bounding box is pre-generated based on the offline map and is used to indicate the area corresponding to the position and size of the escalator in the offline map.

7. The method according to claim 6, characterized in that The escalator bounding box is a rectangular area or a square area, and determining that the current position of the electronic device enters the escalator bounding box according to the camera pose includes: Determine the minimum horizontal coordinate value, the maximum horizontal coordinate value, the minimum vertical coordinate value, and the maximum vertical coordinate value among the coordinate values ​​of the four vertices of the escalator bounding box; When the horizontal coordinate value of the camera posture is less than the maximum horizontal coordinate value and greater than the minimum horizontal coordinate value, and the vertical coordinate value of the camera posture is less than the maximum vertical coordinate value and greater than the minimum vertical coordinate value, it is determined that the current position of the electronic device enters the escalator bounding box.

8. The method according to claim 6, characterized in that The escalator bounding box is a rectangular area or a square area, and determining that the current position of the electronic device enters the escalator bounding box according to the camera pose includes: Determine the horizontal coordinate value and the vertical coordinate value of the center point of the escalator bounding box; When the absolute value of the difference between the horizontal coordinate value of the camera posture and the horizontal coordinate value of the center point is less than a first difference threshold, and the absolute value of the difference between the vertical coordinate value of the camera posture and the vertical coordinate value of the center point is less than a second difference threshold, it is determined that the current position of the electronic device enters the escalator bounding box.

9. The method according to claim 6, characterized in that The escalator bounding box is a circular area, and determining, according to the camera pose, that the current position of the electronic device enters the escalator bounding box includes: Determine the horizontal coordinate value and the vertical coordinate value of the center point of the escalator bounding box; When the absolute value of the difference between the horizontal coordinate value of the camera posture and the horizontal coordinate value of the center point is less than the third difference threshold, and the absolute value of the difference between the vertical coordinate value of the camera posture and the vertical coordinate value of the center point is also less than the third difference threshold, it is determined that the current position of the electronic device enters the escalator bounding box, and the third difference threshold is the radius value of the circular area.

10. The method according to any one of claims 6 to 9, characterized in that: The method further comprises: Performing target detection on the offline map using a target detection algorithm to obtain a target object, where the target object is an escalator; determining the position and size of the target object; The escalator bounding box is generated according to the position and size of the target object.

11. The method according to any one of claims 1 to 10, characterized in that Determining that the current position of the electronic device is at a preset position includes: Performing target detection on the second image using a target detection algorithm to obtain a detection result; If the detection result includes the escalator, it is determined that the current position of the electronic device is at the preset position.

12. The method according to any one of claims 1 to 11, characterized in that After stopping the positioning function, the method further includes: Stop the initialization operation related to the positioning function.

13. An electronic device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 12.

14. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

15. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.

Images