Navigation positioning method and terminal device

Abstract

The present application provides a navigation and positioning method and a terminal device, which are applied to the field of terminal technology. The navigation and positioning method includes: using multiple groups of IMU data obtained through IMU detection, the speed of the terminal device at the current moment obtained through speed measurement unit detection, and disparity data obtained based on multiple first images and second images to determine whether the terminal device is in a stationary state between the first moment and the current moment; if the terminal device is in a stationary state between the first moment and the current moment, determining the position of the terminal device and a third image at the current moment, and displaying the third image through an AR navigation interface, which is conducive to improving the robustness of the zero-speed update scheme, meeting the needs of users in some special scenarios such as uniform motion, large acceleration fluctuations, and the terminal device being blocked, thereby improving the user experience.

Description

Technical Field

[0001] The present application relates to the field of terminal technology, and in particular to a navigation and positioning method and a terminal device. Background Art

[0002] When a user uses the navigation application in the terminal device for navigation and positioning, the terminal device needs to integrate the angular velocity and acceleration measured by the inertial measurement unit (IMU) multiple times to update and correct the current positioning point. When using the augmented reality (AR) navigation mode, since the user may be stationary for a long time, such as stopping to watch, the IMU data may have long-term measurement error accumulation, causing the terminal device's position in the navigation application to deviate from its actual position, and needs to be optimized through a zero-speed update solution.

[0003] The zero-speed update solution is to use the observation equation to update and correct the position of the terminal device in the navigation application after determining that the terminal device is currently stationary. At present, the zero-speed update solution mainly includes two solutions: lightweight accurate robust visual inertial odometry (LARVIO) and open visual inertial navigation system (OPENVINS). Among them, the LARVIO solution uses the observation equation with speed, displacement and angle (VPQ) as the observation value to update and correct the position of the terminal device in the navigation application after judging that the terminal device is currently stationary based on visual data; the OPENVINS solution uses the observation equation with angular velocity and acceleration (ωA) as the observation value to update and correct the position of the terminal device in the navigation application after judging that the terminal device is currently stationary through IMU data or visual data.

[0004] However, the existing zero-speed update solution has low robustness and cannot meet the needs of users in some special scenarios such as uniform motion, large acceleration fluctuations, and terminal devices being blocked, resulting in a poor user experience. Summary of the invention

[0005] The present application provides a navigation and positioning method and a terminal device, which are conducive to improving the robustness of the zero-speed update solution, meeting the needs of users in some special scenarios such as uniform motion, large acceleration fluctuations, and terminal devices being blocked, thereby improving the user experience.

[0006] In a first aspect, a navigation and positioning method is provided, including: continuously performing detection by the IMU between a first moment and a current moment to obtain multiple groups of IMU data; performing detection by the speed measuring unit at the current moment to obtain the speed of the terminal device at the current moment; when the number of the multiple groups of IMU data is greater than or equal to a first threshold, obtaining disparity data based on multiple first images and a second image corresponding to the position of the terminal device at the first moment, the multiple first images are continuously collected by the camera between the first moment and the current moment; using the multiple groups of IMU data, the speed of the terminal device at the current moment and the disparity data, determining whether the terminal device is in a stationary state between the first moment and the current moment; if the terminal device is in a stationary state between the first moment and the current moment, determining the position of the terminal device at the current moment and a third image corresponding to the position of the terminal device at the current moment, and displaying the third image through an AR navigation interface.

[0007] The embodiment of the present application uses multiple groups of IMU data obtained by continuous detection by the IMU between the first moment and the current moment, the speed of the terminal device obtained by the speed measurement unit at the current moment, and the parallax data to judge whether the terminal device is in a stationary state between the first moment and the current moment, and then determines the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment, and displays the third image through the AR navigation interface, refines the judgment conditions of the stationary state, and can more accurately distinguish different scenarios when the user uses the AR navigation mode, avoids the terminal device from mistakenly using the zero-speed update scheme for navigation and positioning, and thus increases the robustness of the zero-speed update scheme, meets the needs of users in some special scenarios such as uniform motion, large acceleration fluctuations, and the terminal device is blocked, thereby improving the user experience.

[0008] In combination with the first aspect, in certain implementations of the first aspect, the executor of the above-mentioned navigation and positioning method may be a terminal device that supports an augmented reality (AR) navigation mode, and the terminal device is provided with an inertial measurement unit (IMU), a speed measurement unit, and a camera.

[0009] It should be understood that between the time when the above-mentioned application is started and the time when the above-mentioned AR navigation mode is started, and between the time when the above-mentioned AR navigation mode is exited and the time when the above-mentioned application is exited, multiple sets of IMU data obtained by continuous detection through the IMU between the time when the above-mentioned application is started and the time when the above-mentioned AR navigation mode is started are integrated, and multiple sets of IMU data obtained by continuous detection through the IMU between the time when the above-mentioned AR navigation mode is exited and the time when the above-mentioned application is exited are integrated to obtain the position of the terminal device.

[0010] It should also be understood that, between the time when the AR navigation mode is started and the time when the AR navigation mode is exited, if the terminal device is not in a stationary state, it is determined whether the terminal device satisfies the visual update condition between the first moment and the current moment; if the terminal device satisfies the visual update condition between the first moment and the current moment, the visual update is used to determine the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment, and the third image is displayed through the AR navigation interface; if the terminal device does not meet the visual update condition between the first moment and the current moment, multiple sets of IMU data obtained by continuous detection by the IMU between the time when the AR navigation mode is started and the time when the AR navigation mode is exited are integrated to determine the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment, and the third image is displayed through the AR navigation interface.

[0011] In combination with the first aspect, in certain implementations of the first aspect, the multiple groups of IMU data include multiple first accelerations and multiple first angular velocities, and the disparity data include an average disparity and a disparity ratio; the use of the multiple groups of IMU data, the speed of the terminal device at the current moment, and the disparity data to determine whether the terminal device is in a stationary state between the first moment and the current moment includes: based on the multiple first accelerations, the multiple first angular velocities, the speed of the terminal device at the current moment, the average disparity and the disparity ratio, determining whether the terminal device is in a stationary state between the first moment and the current moment.

[0012] It should be understood that the above-mentioned first acceleration is the modulus of the three-axis acceleration detected by the IMU in the terminal device coordinate system. Similarly, the above-mentioned first angular velocity is the modulus of the three-axis angular velocity detected by the IMU in the terminal device coordinate system.

[0013] It should also be understood that there is a corresponding relationship between the above-mentioned multiple first accelerations and the multiple first angular velocities.

[0014] The embodiment of the present application uses multiple groups of IMU data obtained by continuous detection by the IMU between the first moment and the current moment, the speed of the terminal device obtained by the speed measurement unit at the current moment, and the parallax data to judge whether the terminal device is in a stationary state between the first moment and the current moment, and then determines the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment, and displays the third image through the AR navigation interface, refines the judgment conditions of the stationary state, and can more accurately distinguish different scenarios when the user uses the AR navigation mode, avoids the terminal device from mistakenly using the zero-speed update scheme for navigation and positioning, and thus increases the robustness of the zero-speed update scheme, meets the needs of users in some special scenarios such as uniform motion, large acceleration fluctuations, and the terminal device is blocked, thereby improving the user experience.

[0015] In combination with the first aspect, in some implementations of the first aspect, the parallax ratio includes a first parallax ratio and a second parallax ratio, the first parallax ratio being the ratio of the number of parallaxes greater than or equal to the second threshold in the number of the multiple parallaxes, and the second parallax ratio being the ratio of the number of parallaxes greater than or equal to the third threshold in the number of the multiple parallaxes; the judging whether the terminal device is in a stationary state between the first moment and the current moment includes: determining that the terminal device is in a stationary state between the first moment and the current moment when the following conditions are met: the multiple first accelerations are all less than or equal to the fourth threshold, the multiple first angular velocities are all less than or equal to the fifth threshold, the average parallax is less than or equal to the sixth threshold, the first parallax ratio is less than or equal to the seventh threshold, the second parallax ratio is less than or equal to the eighth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

[0016] It should be understood that when the plurality of first accelerations are not all less than or equal to the fourth threshold and the plurality of first angular velocities are not all less than or equal to the fifth threshold, any one of the following three conditions may be met:

[0017] Condition 1: at least one of the plurality of first accelerations is greater than or equal to the fourth threshold, and the plurality of first angular velocities are all less than or equal to the fifth threshold.

[0018] Condition 2: the plurality of first accelerations are all less than or equal to the fourth threshold, and at least one of the plurality of first angular velocities is greater than or equal to the fifth threshold.

[0019] Condition 3: at least one first acceleration among the plurality of first accelerations is greater than or equal to the fourth threshold, and at least one first angular velocity among the plurality of first angular velocities is greater than or equal to the fifth threshold.

[0020] Among them, when condition 2 or condition 3 is met, it will continue to be determined whether the terminal device meets the visual update condition between the first moment and the current moment.

[0021] It should also be understood that when the conditions that the average parallax is less than or equal to the sixth threshold, the first parallax proportion is less than or equal to the seventh threshold, and the second parallax proportion is less than or equal to the eighth threshold are not met, or when the speed of the terminal device at the current moment is not less than or equal to the ninth threshold is not met, it will continue to be determined whether the terminal device meets the visual update condition between the first moment and the current moment.

[0022] In this way, the embodiments of the present application can prevent the terminal device from mistakenly using the zero-speed update scheme for navigation and positioning in special scenarios where the user is performing uniform motion by limiting the average parallax and the parallax ratio. At the same time, by limiting the speed of the terminal device at the current moment, it prevents the terminal device from using the zero-speed update scheme for navigation and positioning when it has a higher speed, thereby increasing the robustness of the zero-speed update scheme, meeting the needs of users in special scenarios where the user is performing uniform motion, and thus improving the user experience.

[0023] In combination with the first aspect, in certain implementations of the first aspect, the disparity data also includes: an average feature matching number; the judging whether the terminal device is in a stationary state between the first moment and the current moment also includes: determining that the terminal device is in a stationary state between the first moment and the current moment when the following conditions are met: at least one first acceleration among the multiple first accelerations is greater than or equal to the fourth threshold, the multiple first angular velocities are all less than or equal to the fifth threshold, the average feature matching number is greater than or equal to the tenth threshold, the average disparity is less than or equal to the eleventh threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold; or, at least one first acceleration among the multiple first accelerations is greater than or equal to the fourth threshold, the multiple first angular velocities are all less than or equal to the fifth threshold, the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold, the average disparity is less than or equal to the thirteenth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

[0024] It should be understood that when at least one of the multiple first accelerations is greater than or equal to the fourth threshold and the multiple first angular velocities are all less than or equal to the fifth threshold, any one of the following three conditions may be met:

[0025] Condition 1: the plurality of first accelerations are all less than or equal to the fourth threshold, and the plurality of first angular velocities are all less than or equal to the fifth threshold.

[0026] Condition 2: at least one first acceleration among the plurality of first accelerations is greater than or equal to the fourth threshold, and at least one first angular velocity among the plurality of first angular velocities is greater than or equal to the fifth threshold.

[0027] Condition 3: the plurality of first accelerations are all less than or equal to the fourth threshold, and at least one first angular velocity among the plurality of first angular velocities is greater than or equal to the fifth threshold.

[0028] Among them, when condition 2 or condition 3 is met, it will continue to be determined whether the terminal device meets the visual update condition between the first moment and the current moment.

[0029] It should also be understood that when the conditions that the average feature matching number is greater than or equal to the tenth threshold and the average disparity is less than or equal to the eleventh threshold are not satisfied, it will continue to be judged whether the conditions that the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold and the average disparity is less than or equal to the thirteenth threshold are satisfied. When the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold and the average disparity is less than or equal to the thirteenth threshold, it will be judged whether the speed of the terminal device at the current moment is less than or equal to the ninth threshold. When the conditions that the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold and the average disparity is less than or equal to the thirteenth threshold are not satisfied, or when the speed of the terminal device at the current moment is less than or equal to the ninth threshold is not satisfied, it will continue to be judged whether the terminal device meets the visual update condition between the first moment and the current moment.

[0030] It should be understood that the eleventh threshold value and the sixth threshold value may be equal or different, and this application does not limit this. The thirteenth threshold value and the sixth threshold value may be equal or different, and this application does not limit this. The twelfth threshold value should be greater than or equal to the tenth threshold value.

[0031] In this way, the embodiment of the present application can prevent the terminal device from mistakenly using the zero-speed update scheme for navigation positioning in special scenarios where the user is performing a small-range slow rotation by limiting the first angular velocity, thereby increasing the robustness of the zero-speed update scheme, meeting the needs of users in special scenarios of uniform motion, and thus improving the user experience.

[0032] In combination with the first aspect, in some implementations of the first aspect, the parallax ratio includes a first parallax ratio and a second parallax ratio, the first parallax ratio being the ratio of the number of parallaxes greater than or equal to the second threshold in the number of the multiple parallaxes, and the second parallax ratio being the ratio of the number of parallaxes greater than or equal to the third threshold in the number of the multiple parallaxes; the method also includes: based on the multiple first accelerations and the multiple first angular velocities, obtaining multiple second accelerations and multiple second angular velocities; the judging whether the terminal device is in a stationary state between the first moment and the current moment includes: determining that the terminal device is in a stationary state between the first moment and the current moment when the following conditions are met: the multiple second accelerations are all less than or equal to the fourth threshold, the multiple second angular velocities are all less than or equal to the fifth threshold, the average parallax is less than or equal to the sixth threshold, the first parallax ratio is less than or equal to the seventh threshold, the second parallax ratio is less than or equal to the eighth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

[0033] It should be understood that there is also a corresponding relationship between the multiple second accelerations and the multiple second angular velocities, which is similar to the corresponding relationship between the multiple first accelerations and the multiple first angular velocities, and will not be elaborated here.

[0034] In combination with the first aspect, in certain implementations of the first aspect, the disparity data also includes: an average feature matching number; the judging whether the terminal device is in a stationary state between the first moment and the current moment also includes: determining that the terminal device is in a stationary state between the first moment and the current moment when the following conditions are met: at least one second acceleration among the multiple second accelerations is greater than or equal to the fourth threshold, the multiple second angular velocities are all less than or equal to the fifth threshold, the average feature matching number is greater than or equal to the tenth threshold, the average disparity is less than or equal to the eleventh threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold; or, at least one second acceleration among the multiple second accelerations is greater than or equal to the fourth threshold, the multiple second angular velocities are all less than or equal to the fifth threshold, the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold, the average disparity is less than or equal to the thirteenth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

[0035] In this way, the embodiment of the present application can obtain multiple second accelerations and multiple second angular velocities by optimizing multiple groups of IMU data respectively, and use multiple second accelerations, multiple second angular velocities, the speed of the terminal device at the current moment, the average parallax and the parallax ratio to determine whether the terminal device is in a stationary state between the first moment and the current moment, thereby compensating for the measurement error of the IMU and increasing the accuracy of the judgment result.

[0036] In combination with the first aspect, in certain implementations of the first aspect, the multiple groups of IMU data also include multiple gravitational accelerations; the multiple second accelerations and multiple second angular velocities are obtained based on the multiple first accelerations and the multiple first angular velocities, including: calculating the absolute value of each first acceleration in the multiple first accelerations and the difference between the gravitational acceleration corresponding to each first acceleration to obtain multiple absolute values, and determining the multiple absolute values ​​as the multiple second accelerations; and determining the multiple first angular velocities as the multiple second angular velocities.

[0037] In this way, the embodiment of the present application can reduce the judgment result error caused by the inaccurate measurement of the acceleration sensor in the IMU, and at the same time, it can also avoid the terminal device from being inaccurate when the gravity acceleration is not 9.81m / s 2 In special scenarios where the zero-speed update solution cannot be used for navigation and positioning, the robustness of the zero-speed update solution is increased, meeting the needs of users in special scenarios of uniform motion, thereby improving the user experience.

[0038] In combination with the first aspect, in certain implementations of the first aspect, if the terminal device is in a stationary state between the first moment and the current moment, determining the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment, includes: if the terminal device is in a stationary state between the first moment and the current moment, determining an update parameter based on whether the terminal device moves between the startup moment of the AR navigation application to the current moment and the number of the multiple first images; constructing an equation based on the update parameter to obtain the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment.

[0039] It should be understood that whether the terminal device moves between the startup time of the above-mentioned AR navigation application and the current time can also be understood as the initialization state of the AR navigation application. Specifically, the initialization state of the AR navigation application corresponding to the movement of the terminal device is dynamic initialization, and the initialization state of the AR navigation application corresponding to the non-movement of the terminal device is static initialization.

[0040] In combination with the first aspect, in certain implementations of the first aspect, the update parameters are determined based on whether the terminal device has moved between the startup moment of the AR navigation application and the current moment and the number of the multiple first images, including: when the terminal device has not moved between the startup moment and the current moment and the number of the multiple first images is greater than or equal to a fourteenth threshold, the speed, displacement and angle between two adjacent first images in the acquisition sequence, and the speed, displacement and angle between the second image and the first image acquired at the first moment are determined as the update parameters.

[0041] It should be understood that the filtering algorithm includes many types, and the embodiments of the present application do not limit the specific filtering algorithm used. Exemplarily, the filtering algorithm may be an extended Kalman filter (EKF) algorithm.

[0042] In combination with the first aspect, in certain implementations of the first aspect, the update parameters are determined based on whether the terminal device moves between the startup moment of the AR navigation application and the current moment and the number of the multiple first images, and also include: when the terminal device moves between the startup moment and the current moment, or when the number of the multiple first images is less than or equal to a fourteenth threshold, the speed between two adjacent first images in the acquisition sequence and the speed between the second image and the first image acquired at the first moment are determined as the update parameters.

[0043] It should be understood that the terminal device moves between the start time and the current time, or the number of the plurality of first images is less than or equal to the fourteenth threshold, specifically including the following three situations:

[0044] Case 1: The terminal device moves between the start time and the current time, and the number of the multiple first images is less than or equal to the fourteenth threshold.

[0045] Case 2: The terminal device moves between the start time and the current time, and the number of the multiple first images is greater than or equal to the fourteenth threshold.

[0046] Case 3: The terminal device does not move between the start time and the current time, and the number of the plurality of first images is less than or equal to the fourteenth threshold.

[0047] In this way, the embodiment of the present application can determine different update parameters according to different scenarios, and then determine the location of the terminal device, thereby improving the accuracy of navigation positioning and the robustness of the zero-speed update solution, thereby improving the user experience.

[0048] In combination with the first aspect, in certain implementations of the first aspect, constructing an equation based on the updated parameters to obtain the position of the terminal device at the current moment and a third image corresponding to the position of the terminal device at the current moment includes: solving the equation using a filtering algorithm to obtain the position of the terminal device at the current moment and a third image corresponding to the position of the terminal device at the current moment.

[0049] In this way, the embodiment of the present application can determine different update parameters according to different scenarios, and then determine the location of the terminal device, thereby improving the accuracy of navigation positioning and the robustness of the zero-speed update solution, thereby improving the user experience.

[0050] In combination with the first aspect, in certain implementations of the first aspect, the disparity data is obtained based on multiple first images and a second image corresponding to the position of the terminal device at the first moment, including: obtaining multiple feature matching numbers and multiple disparities based on the multiple first images and the second image; determining the average value of the multiple feature matching numbers as the average feature matching number; determining the average value of the multiple disparities as the average disparity; determining the ratio of the number of disparities in the multiple disparities that are greater than or equal to the second threshold to the number of the multiple disparities as the first disparity ratio; determining the ratio of the number of disparities in the multiple disparities that are greater than or equal to the third threshold to the number of the multiple disparities as the second disparity ratio.

[0051] In combination with the first aspect, in certain implementations of the first aspect, obtaining multiple feature matching numbers and multiple disparities based on the multiple first images and the second images includes: using a feature extraction algorithm to extract features from the multiple first images and the second images respectively to obtain multiple features of each first image in the multiple first images and multiple features of the second image; matching multiple features of two first images adjacent in acquisition order respectively, and matching multiple features of the second image with multiple features of the first image acquired at the first moment respectively to obtain multiple groups of matching features; determining the number of matching features in each group of the multiple groups of matching features as the multiple feature matching numbers; and determining the average matching distance of multiple matching features in each group of matching features as the multiple disparities.

[0052] It should be understood that there are many types of feature extraction algorithms, and the embodiments of the present application do not limit the specific feature extraction algorithm used. Exemplarily, the feature extraction algorithm may be a sparse optical flow (Kanade-Lucas-Tomasi, KTL) algorithm.

[0053] In this way, the embodiment of the present application determines whether the terminal device is in a stationary state between the first moment and the current moment by using the second image corresponding to the position of the terminal device at the first moment and the disparity data between multiple first images continuously collected by the camera between the first moment and the current moment. This can avoid the terminal device from mistakenly using the zero-speed update scheme for navigation and positioning in special scenarios where the user is moving or the waves are peaking and troughing, thereby increasing the robustness of the zero-speed update scheme, meeting the needs of users in special scenarios where they are moving, and thus improving the user experience.

[0054] In a second aspect, a terminal device is provided, comprising: an inertial measurement unit IMU, a speed measuring unit, a camera and a processor, wherein the processor is coupled to a memory, and the memory is used to store a computer program. When the processor calls the computer program, the terminal device executes the method described in the first aspect or any possible implementation of the first aspect through the IMU, the speed measuring unit, and the camera.

[0055] In a third aspect, a processor is provided, comprising: an input circuit, an output circuit, and a processing circuit. The processing circuit is used to receive a signal through the input circuit and transmit a signal through the output circuit, so that the processor executes the method in the first aspect or any possible implementation of the first aspect.

[0056] In the specific implementation process, the processor can be a chip, the input circuit can be an input pin, the output circuit can be an output pin, and the processing circuit can be a transistor, a gate circuit, a trigger, and various logic circuits. The input signal received by the input circuit can be, for example, but not limited to, received and input by a receiver, and the signal output by the output circuit can be, for example, but not limited to, output to a transmitter and transmitted by the transmitter, and the input circuit and the output circuit can be the same circuit, which is used as an input circuit and an output circuit at different times. The embodiments of the present application do not limit the specific implementation methods of the processor and various circuits.

[0057] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which is used to store a computer program, and the computer program includes instructions for implementing the method described in the first aspect or any possible implementation of the first aspect.

[0058] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program code. When the computer program code runs on a computer, the computer implements the method described in the first aspect or any possible implementation manner of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Figure 1 A schematic diagram of the structure of a terminal device provided in an embodiment of the present application;

[0060] Figure 2 A schematic diagram of the software structure of a terminal device provided in an embodiment of the present application;

[0061] Figure 3 A schematic flowchart of a navigation and positioning method provided in an embodiment of the present application;

[0062] Figure 4 A schematic flow chart of another navigation and positioning method provided in an embodiment of the present application;

[0063] Figure 5 A schematic flow chart of another navigation and positioning method provided in an embodiment of the present application;

[0064] Figure 6 A schematic flow chart of another navigation and positioning method provided in an embodiment of the present application;

[0065] Figure 7 A schematic block diagram of a terminal device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The technical solution in this application will be described below in conjunction with the accompanying drawings.

[0067] In order to clearly describe the technical solutions of the embodiments of the present application, in the embodiments of the present application, words such as "first" and "second" are used to distinguish the same or similar items with substantially the same functions and effects. Those skilled in the art can understand that words such as "first" and "second" do not limit the quantity and execution order, and words such as "first" and "second" do not necessarily limit the difference.

[0068] It should be noted that in the embodiments of the present application, words such as "exemplarily" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplarily" or "for example" in the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplarily" or "for example" is intended to present related concepts in a specific way.

[0069] In the embodiments of the present application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, a--c, bc, or abc, where a, b, c can be single or multiple.

[0070] In order to better understand the embodiments of the present application, the terminal device involved in the embodiments of the present application is introduced below.

[0071] Figure 1 A schematic structural diagram of the terminal device 100 is shown.

[0072] The terminal device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, a subscriber identification module (SIM) card interface 195, a speed measurement unit 196, etc. Among them, the sensor module 180 may include a pressure sensor 180A, an inertial measurement unit (IMU) 180B, an air pressure sensor 180C, a magnetic sensor 180D, a bone conduction sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, etc. Among them, the IMU 180B may include a gyroscope sensor 180B1 and an acceleration sensor 180B2.

[0073] It is understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0074] The processor 110 may include one or more processing units, for example, the processor 110 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0075] The controller can generate operation control signals according to the instruction operation code and timing signal to complete the control of instruction fetching and execution.

[0076] The processor 110 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. The memory may store instructions or data that the processor 110 has just used or cyclically used. If the processor 110 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0077] In some embodiments, the processor 110 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0078] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple groups of I2C buses. The processor 110 may be coupled to the touch sensor 180K, the charger, the flash, the camera 193, etc. through different I2C bus interfaces. For example: the processor 110 may be coupled to the touch sensor 180K through the I2C interface, so that the processor 110 communicates with the touch sensor 180K through the I2C bus interface, thereby realizing the touch function of the terminal device 100.

[0079] The I2S interface can be used for audio communication. In some embodiments, the processor 110 can include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to achieve communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 via the I2S interface to achieve the function of answering a call through a Bluetooth headset.

[0080] The PCM interface can also be used for audio communication, sampling, quantizing and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 160 can be coupled via a PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 160 via the PCM interface to implement the function of answering calls via a Bluetooth headset. Both the above-mentioned I2S interface and the above-mentioned PCM interface can be used for audio communication.

[0081] The UART interface is a universal serial data bus for asynchronous communication. The bus can be a bidirectional communication bus. It converts the data to be transmitted between serial communication and parallel communication. In some embodiments, the UART interface is generally used to connect the processor 110 and the wireless communication module 160. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 160 through the UART interface to implement the Bluetooth function. In some embodiments, the audio module 170 can transmit an audio signal to the wireless communication module 160 through the UART interface to implement the function of playing music through a Bluetooth headset.

[0082] The MIPI interface can be used to connect the processor 110 with peripheral devices such as the display screen 194 and the camera 193. The MIPI interface includes a camera serial interface (CSI), a display serial interface (DSI), etc. In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to implement the shooting function of the terminal device 100. The processor 110 and the display screen 194 communicate via the DSI interface to implement the display function of the terminal device 100.

[0083] The GPIO interface can be configured by software. The GPIO interface can be configured as a control signal or as a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 with the camera 193, the display 194, the wireless communication module 160, the audio module 170, the sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0084] The USB interface 130 is an interface that complies with the USB standard specification, and specifically can be a Mini USB interface, a Micro USB interface, a USB Type C interface, etc. The USB interface 130 can be used to connect a charger to charge the terminal device 100, and can also be used to transmit data between the terminal device 100 and peripheral devices. It can also be used to connect headphones to play audio through the headphones. The interface can also be used to connect other terminal devices, such as AR devices, etc.

[0085] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the terminal device 100. In other embodiments of the present application, the terminal device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0086] The charging management module 140 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 may receive charging input from a wired charger through the USB interface 130. In some wireless charging embodiments, the charging management module 140 may receive wireless charging input through a wireless charging coil of the terminal device 100. While the charging management module 140 is charging the battery 142, it may also power the terminal device through the power management module 141.

[0087] The power management module 141 is used to connect the battery 142, the charging management module 140 and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, the internal memory 121, the display screen 194, the camera 193, and the wireless communication module 160. The power management module 141 can also be used to monitor parameters such as battery capacity, battery cycle number, battery health status (leakage, impedance), etc. In some other embodiments, the power management module 141 can also be set in the processor 110. In other embodiments, the power management module 141 and the charging management module 140 can also be set in the same device.

[0088] The wireless communication function of the terminal device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0089] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0090] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the terminal device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0091] The modem processor may include a modulator and a demodulator. Among them, the modulator is used to modulate the low-frequency baseband signal to be sent into a medium-high frequency signal. The demodulator is used to demodulate the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After the low-frequency baseband signal is processed by the baseband processor, it is passed to the application processor. The application processor outputs a sound signal through an audio device (not limited to a speaker 170A, a receiver 170B, etc.), or displays an image or video through a display screen 194. In some embodiments, the modem processor may be an independent device. In other embodiments, the modem processor may be independent of the processor 110 and be set in the same device as the mobile communication module 150 or other functional modules.

[0092] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the terminal device 100. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and performs filtering, and sends the processed signal to the processor 110. The wireless communication module 160 can also receive the signal to be sent from the processor 110, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0093] In some embodiments, the antenna 1 of the terminal device 100 is coupled to the mobile communication module 150, and the antenna 2 is coupled to the wireless communication module 160, so that the terminal device 100 can communicate with the network and other devices through wireless communication technology. The above wireless communication technology may include global system for mobile communications (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), time-division code division multiple access (TD-SCDMA), long term evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technology, etc. The above-mentioned GNSS may include a global positioning system (GPS), a global navigation satellite system (GLONASS), a Beidou navigation satellite system (BDS), a quasi-zenith satellite system (QZSS) and / or a satellite based augmentation system (SBAS).

[0094] The terminal device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0095] The display screen 194 is used to display images, videos, etc. The display screen 194 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicroLed, Micro-oLed, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the terminal device 100 may include 1 or N display screens 194, where N is a positive integer greater than 1.

[0096] The terminal device 100 can realize the shooting function through ISP, camera 193, video codec, GPU, display screen 194 and application processor.

[0097] ISP is used to process the data fed back by camera 193. For example, when taking a photo, the shutter is opened, and the light is transmitted to the camera photosensitive element through the lens. The light signal is converted into an electrical signal, and the camera photosensitive element transmits the electrical signal to ISP for processing and converts it into an image visible to the naked eye. ISP can also perform algorithm optimization on the noise, brightness, and skin color of the image. ISP can also optimize the exposure, color temperature and other parameters of the shooting scene. In some embodiments, ISP can be set in camera 193.

[0098] The camera 193 is used to capture still images or videos. The object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the optical signal into an electrical signal, and then passes the electrical signal to the ISP to be converted into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into an image signal in a standard RGB, YUV or other format. In some embodiments, the terminal device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0099] The digital signal processor is used to process digital signals, and can process not only digital image signals but also other digital signals. For example, when the terminal device 100 is selecting a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy.

[0100] Video codecs are used to compress or decompress digital videos. The terminal device 100 may support one or more video codecs. Thus, the terminal device 100 may play or record videos in various coding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0101] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of the terminal device 100 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0102] The external memory interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the terminal device 100. The external memory card communicates with the processor 110 through the external memory interface 120 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0103] The internal memory 121 can be used to store computer executable program codes, and the above executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the terminal device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the terminal device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0104] The terminal device 100 can implement audio functions such as music playing and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0105] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be arranged in the processor 110, or some functional modules of the audio module 170 can be arranged in the processor 110.

[0106] The speaker 170A, also called a "speaker", is used to convert an audio electrical signal into a sound signal. The terminal device 100 can listen to music or listen to a hands-free call through the speaker 170A.

[0107] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the terminal device 100 receives a call or voice message, the voice can be received by placing the receiver 170B close to the ear.

[0108] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The terminal device 100 can be provided with at least one microphone 170C. In other embodiments, the terminal device 100 can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the terminal device 100 can also be provided with three, four or more microphones 170C to collect sound signals, reduce noise, identify the source of sound, realize directional recording function, etc.

[0109] The earphone interface 170D is used to connect a wired earphone and can be a USB interface 130, or a 3.5 mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0110] The pressure sensor 180A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 194. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. The capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The terminal device 100 determines the intensity of the pressure according to the change in capacitance. When a touch operation acts on the display screen 194, the terminal device 100 detects the above touch operation intensity according to the pressure sensor 180A. The terminal device 100 can also calculate the touch position according to the detection signal of the pressure sensor 180A. In some embodiments, touch operations acting on the same touch position but with different touch operation intensities can correspond to different operation instructions. For example: when a touch operation with a touch operation intensity less than the first pressure threshold acts on the short message application icon, an instruction to view the short message is executed. When a touch operation with a touch operation intensity greater than or equal to the first pressure threshold acts on the short message application icon, an instruction to create a new short message is executed.

[0111] The gyroscope sensor 180B1 can be used to determine the motion posture of the terminal device 100. In some embodiments, the angular velocity of the terminal device 100 around three axes (i.e., x, y, and z axes) can be determined by the gyroscope sensor 180B1. The gyroscope sensor 180B1 can be used for anti-shake shooting. For example, when the shutter is pressed, the gyroscope sensor 180B1 detects the angle of the terminal device 100 shaking, calculates the distance that the lens module needs to compensate based on the angle, and allows the lens to offset the shaking of the terminal device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B1 can also be used for navigation and somatosensory game scenes.

[0112] The acceleration sensor 180B2 can detect the magnitude of the acceleration of the terminal device 100 in various directions (generally three axes). When the terminal device 100 is stationary, the magnitude and direction of gravity can be detected. It can also be used to identify the posture of the terminal device and is applied to applications such as horizontal and vertical screen switching and pedometers.

[0113] The air pressure sensor 180C is used to measure air pressure. In some embodiments, the terminal device 100 calculates the altitude through the air pressure value measured by the air pressure sensor 180C to assist positioning and navigation.

[0114] The magnetic sensor 180D includes a Hall sensor. The terminal device 100 can use the magnetic sensor 180D to detect the opening and closing of the flip leather case. In some embodiments, when the terminal device 100 is a flip phone, the terminal device 100 can detect the opening and closing of the flip cover according to the magnetic sensor 180D. Then, according to the detected opening and closing state of the leather case or the opening and closing state of the flip cover, the flip cover automatic unlocking and other features are set.

[0115] The bone conduction sensor 180E can obtain a vibration signal. In some embodiments, the bone conduction sensor 180E can obtain a vibration signal of a vibrating bone block of the vocal part of the human body. The bone conduction sensor 180E can also contact the human pulse to receive a blood pressure beat signal. In some embodiments, the bone conduction sensor 180E can also be set in an earphone and combined into a bone conduction earphone. The audio module 170 can parse out a voice signal based on the vibration signal of the vibrating bone block of the vocal part obtained by the above-mentioned bone conduction sensor 180E to realize the voice function. The application processor can parse the heart rate information based on the blood pressure beat signal obtained by the above-mentioned bone conduction sensor 180E to realize the heart rate detection function.

[0116] The distance sensor 180F is used to measure the distance. The terminal device 100 can measure the distance by infrared or laser. In some embodiments, when shooting a scene, the terminal device 100 can use the distance sensor 180F to measure the distance to achieve fast focusing.

[0117] The proximity light sensor 180G may include, for example, a light emitting diode (LED) and a light detector, such as a photodiode. The light emitting diode may be an infrared light emitting diode. The terminal device 100 emits infrared light outward through the light emitting diode. The terminal device 100 uses a photodiode to detect infrared reflected light from nearby objects. When sufficient reflected light is detected, it can be determined that there is an object near the terminal device 100. When insufficient reflected light is detected, the terminal device 100 can determine that there is no object near the terminal device 100. The terminal device 100 can use the proximity light sensor 180G to detect that the user holds the terminal device 100 close to the ear to talk, so as to automatically turn off the screen to save power. The proximity light sensor 180G can also be used in leather case mode, and pocket mode automatically unlocks and locks the screen.

[0118] The ambient light sensor 180L is used to sense the ambient light brightness. The terminal device 100 can adaptively adjust the brightness of the display screen 194 according to the perceived ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance when taking pictures. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether the terminal device 100 is in a pocket to prevent accidental touch.

[0119] The fingerprint sensor 180H is used to collect fingerprints. The terminal device 100 can use the collected fingerprint characteristics to achieve fingerprint unlocking, access application locks, fingerprint photography, fingerprint call answering, etc.

[0120] The temperature sensor 180J is used to detect temperature. In some embodiments, the terminal device 100 uses the temperature detected by the temperature sensor 180J to execute a temperature processing strategy. For example, when the temperature reported by the temperature sensor 180J exceeds a threshold, the terminal device 100 reduces the performance of the processor located near the temperature sensor 180J to reduce power consumption and implement thermal protection. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 heats the battery 142 to avoid abnormal shutdown of the terminal device 100 due to low temperature. In other embodiments, when the temperature is lower than another threshold, the terminal device 100 performs a boost on the output voltage of the battery 142 to avoid abnormal shutdown caused by low temperature.

[0121] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194, and the touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the terminal device 100, which is different from the position of the display screen 194.

[0122] The key 190 includes a power key, a volume key, etc. The key 190 may be a mechanical key or a touch key. The terminal device 100 may receive key input and generate key signal input related to user settings and function control of the terminal device 100.

[0123] Motor 191 can generate vibration prompts. Motor 191 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback. For example, touch operations acting on different applications (such as taking pictures, audio playback, etc.) can correspond to different vibration feedback effects. For touch operations acting on different areas of the display screen 194, motor 191 can also correspond to different vibration feedback effects. Different application scenarios (for example: time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also support customization.

[0124] Indicator 192 may be an indicator light, which may be used to indicate charging status, power changes, messages, missed calls, notifications, etc.

[0125] The SIM card interface 195 is used to connect a SIM card. The SIM card can be connected to and separated from the terminal device 100 by inserting it into the SIM card interface 195 or pulling it out from the SIM card interface 195. The terminal device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, and the like. Multiple cards can be inserted into the same SIM card interface 195 at the same time. The types of the above multiple cards can be the same or different. The SIM card interface 195 can also be compatible with different types of SIM cards. The SIM card interface 195 can also be compatible with external memory cards. The terminal device 100 interacts with the network through the SIM card to implement functions such as calls and data communications. In some embodiments, the terminal device 100 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 100 and cannot be separated from the terminal device 100.

[0126] The speed measuring unit 196 is used to detect the speed. In some embodiments, the terminal device 100 uses the speed detected by the speed measuring unit 196 to assist in positioning and navigation.

[0127] The software system of the terminal device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The present application embodiment takes the Android system of the layered architecture as an example to exemplify the software structure of the terminal device 100.

[0128] Figure 2 A software structure block diagram of the terminal device 100 according to an embodiment of the present application is shown.

[0129] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into five layers, from top to bottom: application layer, application framework layer, Android runtime and system library, hardware abstraction layer (HAL) layer and kernel layer.

[0130] The application layer can include a series of application packages.

[0131] like Figure 2 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message and other applications.

[0132] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0133] like Figure 2 As shown, the application framework layer may include a window manager, a content provider, a view system, a phone manager, a resource manager, a notification manager, and the like.

[0134] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0135] Content providers are used to store and retrieve data and make it accessible to applications. The above data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0136] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0137] The phone manager is used to provide communication functions of the terminal device 100, such as management of call status (including connection, disconnection, etc.).

[0138] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0139] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the screen in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, the terminal device vibrates, the indicator light flashes, etc.

[0140] Android Runtime includes core libraries and virtual machines. Android runtime is responsible for scheduling and management of the Android system.

[0141] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0142] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0143] The system library may include multiple functional modules, such as surface manager, media libraries, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0144] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0145] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0146] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0147] A 2D graphics engine is a drawing engine for 2D drawings.

[0148] The HAL layer is an abstract interface layer between the kernel layer and the upper layer, providing a unified interface for the upper layer. The HAL layer can contain multiple functional interfaces. For example: upper layer software interface, internal communication interface, internal Ethernet interface, user access port, etc.

[0149] The application layer can call the HAL layer through the application framework layer, Android Runtime and system library. Specifically, after the application layer detects the information of the user acting on the application in the terminal device, the detected information of the user acting on the application in the terminal device is sent to the HAL layer, and the HAL layer determines the hardware module corresponding to the application based on the received information, and operates the hardware module based on the received information.

[0150] It should be understood that in actual application, the application framework layer, Android Runtime and system library only provide channels for the interaction process between the above application layer and the above HAL layer.

[0151] The kernel layer is the layer between hardware and software. The kernel layer includes at least display driver, camera driver, audio driver, sensor driver, and the sensor driver includes at least IMU driver.

[0152] The terminal device of the embodiment of the present application may be any electronic device equipped with a camera and an IMU, such as a mobile phone, a tablet computer, etc. It should be understood that the embodiment of the present application does not limit the specific technology and specific device form adopted by the terminal device.

[0153] When users use the navigation application in the terminal device for navigation and positioning, the terminal device needs to integrate the angular velocity and acceleration measured by the IMU multiple times to update and correct the current positioning point. When using the augmented reality (AR) navigation mode, since users may be stationary for a long time, such as standing still to watch, the IMU data may have long-term measurement error accumulation, causing the terminal device's position in the navigation application to deviate from its actual position, which needs to be optimized through a zero-speed update solution.

[0154] In a possible implementation, the navigation application in the terminal device is based on the extended Kalman filter (EKF) simultaneous positioning and map construction (simulation localization and mapping, SLAM) solution for navigation and positioning, wherein SLAM may include five modules, namely, a data acquisition module, a visual odometry module, a back-end optimization module, a mapping module, and a loop closure detection module. Specifically, the terminal device uses the visual odometry module to filter and fuse the IMU data and an image sequence (which may also be a video stream) to obtain an estimated pose, and then uses the back-end optimization module to optimize the estimated pose using the EKF to obtain the final pose, and then uses the mapping module to build and locate the final pose after the final pose passes the detection of the loop closure detection module. Since IMU data often has a lot of noise, and there are errors and mismatches in the relevant hardware in the terminal device, the mapping and positioning are not accurate enough, so it is necessary to optimize it through a zero-speed update solution.

[0155] The zero-speed update solution is to use the observation equation to update and correct the position of the terminal device in the navigation application after determining that the terminal device is currently stationary. At present, the zero-speed update solution mainly includes monocular visual inertial odometry (lightweight accurate robust visual inertial odometry, LARVIO) and open source visual inertial navigation system (open visual inertial navigation system The LARVIO scheme determines that the terminal device is currently stationary when the number of tracked feature points in the current image frame is greater than 20 and the matching distance of the 11th maximum feature point is less than the normalized plane threshold 2e-3, and then uses the observation equation with velocity, displacement and angle (VPQ) as observation values ​​to update and correct the position of the terminal device in the navigation application; the OPENVINS scheme determines that the terminal device is currently stationary when any of the following conditions is met: the number of tracked feature points in the current image frame is greater than 20 and the disparity between the current image frame and the next-to-latest image frame is less than the threshold (for example, 1.5 pixels), or the chi-square test value of the IMU data is less than the threshold (for example, 5 times the chi-square theoretical value) and the speed of the next-to-latest frame is less than the threshold (for example, 0.1 meters per second). Then, the observation equation with angular velocity and acceleration (ωA) as observation values ​​is used to update and correct the position of the terminal device in the navigation application.

[0156] However, the existing zero-speed update solution has low robustness and cannot meet the needs of users in some special scenarios such as uniform motion, large acceleration fluctuations, and terminal devices being blocked, resulting in a poor user experience.

[0157] In view of this, the embodiment of the present application determines whether the terminal device is in a stationary state between the first moment and the current moment by using multiple sets of IMU data, the speed and parallax data of the terminal device at the current moment, and refines the judgment conditions of the stationary state. It can more accurately distinguish different scenarios when users use the AR navigation mode, avoid the terminal device from mistakenly using the zero-speed update scheme for navigation and positioning, and thus increase the robustness of the zero-speed update scheme, meet the needs of users in some special scenarios such as uniform motion, large acceleration fluctuations, and the terminal device is blocked, thereby improving the user experience.

[0158] It should be understood that the above special scenarios are only examples. In addition, special scenarios may include terminal device shaking, user creeping, dynamic object interference around the terminal device, user slow rotation in a small range, and the terminal device being in a weak texture area. The specific scenarios of special scenarios can be adjusted according to actual conditions, and shall not constitute any limitation on the implementation process of the embodiments of the present application.

[0159] For better understanding, the following first describes multiple moments and a navigation interface involved in the navigation and positioning method provided in the embodiment of the present application.

[0160] The multiple moments involved in the navigation and positioning method provided in the embodiment of the present application may include: the application startup moment, the AR navigation mode startup moment, the first moment, the current moment, the AR navigation mode exit moment, and the application exit moment. It should be understood that the above multiple moments are only examples. In the actual application process, other moments may also be involved, and the embodiment of the present application is not limited here. The following is a detailed description of the above multiple moments in combination with the interaction process between hardware such as IMU, camera, speed measurement unit and processor.

[0161] 1. Application startup time, that is, the time when the user starts a certain application for navigation and positioning through the terminal device. In response to the user's operation, the IMU starts and continues to detect to obtain the IMU data at different times, the camera starts and continues to collect to obtain the first image at different times, and the speed measurement unit starts and continues to detect to obtain the speed of the terminal device at different times.

[0162] Optionally, the above-mentioned application for navigation and positioning may include multiple navigation modes, wherein the specific types and numbers of navigation modes are not limited in this application. Exemplarily, the above-mentioned navigation modes may include: a normal navigation mode and an AR navigation mode.

[0163] 2. The moment when the AR navigation mode is started, that is, when the user starts the AR navigation mode in one of the above applications through the terminal device. In response to the user's operation, the IMU sends to the processor multiple sets of IMU data continuously detected between the application start-up time and the AR navigation mode start-up time, the camera sends to the processor multiple first images continuously collected between the application start-up time and the AR navigation mode start-up time, and the speed measurement unit sends to the processor the speed of the terminal device detected at the time of the AR navigation mode start-up. Correspondingly, the processor receives the multiple sets of IMU data, the multiple first images, and the speed of the terminal device at the time of the AR navigation mode start-up.

[0164] It should be understood that the specific navigation mode requires the user to launch the application before selecting it, so the AR navigation mode is launched after the application is launched.

[0165] Optionally, the above-mentioned application for navigation and positioning may only include an AR navigation mode. In this case, the AR navigation mode startup time is the application startup time.

[0166] 3. The first moment is a moment when the user uses the AR navigation mode through the terminal device. At this time, the AR navigation interface displays the second image corresponding to the position of the terminal device at the first moment.

[0167] 4. The current moment, the moment after the first moment. At this time, the AR navigation interface still displays the second image corresponding to the position of the terminal device at the first moment, and is about to display the third image corresponding to the position of the terminal device at the current moment. The IMU sends multiple sets of IMU data continuously detected between the first moment and the current moment to the processor, the camera sends multiple first images continuously collected between the first moment and the current moment to the processor, and the speed measurement unit sends the detected speed of the terminal device at the current moment to the processor. Correspondingly, the processor receives the multiple sets of IMU data, the multiple first images, and the speed of the terminal device at the current moment.

[0168] 5. AR navigation mode exit moment, that is, when the user exits the AR navigation mode in the above application through the terminal device. In response to the user's operation, the IMU stops sending multiple sets of IMU data to the processor, the camera stops sending multiple first images to the processor, and the speed measurement unit stops sending the speed of the terminal device to the processor.

[0169] 6. Application exit time, that is, the time when the user exits a navigation and positioning application through the terminal device. In response to the user's operation, the IMU stops detecting, the camera stops collecting, and the speed measurement unit stops detecting.

[0170] It should be understood that exiting the AR navigation mode or switching from the AR navigation mode to other navigation modes requires the user to make a selection during the application running process. Therefore, the application exit time is after the AR navigation mode exit time.

[0171] Optionally, the user can exit the application directly while using the AR navigation mode. In this case, the application exit time is the AR navigation mode exit time.

[0172] Figure 3 A schematic flow chart of a navigation positioning method 300 provided in an embodiment of the present application is shown. The method 300 can be applied to a terminal device supporting an AR navigation mode, the terminal device being provided with an IMU, a speed measurement unit and a camera. The hardware structure of the terminal device can be as follows: Figure 1 As shown, the software structure of the terminal device can be as follows Figure 2 As shown, but the embodiments of the present application are not limited thereto. Figure 3 As shown, the method 300 may include the following steps:

[0173] S301, continuously perform detection by IMU between a first moment and a current moment to obtain multiple sets of IMU data.

[0174] S302: Detect at the current moment through a speed measuring unit to obtain the speed of the terminal device at the current moment.

[0175] S303. When the number of multiple groups of IMU data is greater than or equal to a first threshold, disparity data is obtained based on multiple first images and a second image corresponding to the position of the terminal device at a first moment, and the multiple first images are continuously collected by the camera between the first moment and the current moment.

[0176] It should be understood that the above S301 and S302 are not limited in sequence.

[0177] S304, using multiple sets of IMU data, the speed and parallax data of the terminal device at the current moment, to determine whether the terminal device is in a stationary state between the first moment and the current moment.

[0178] S305: If the terminal device is in a stationary state between the first moment and the current moment, determine the position of the terminal device at the current moment and a third image corresponding to the position of the terminal device at the current moment, and display the third image through the AR navigation interface.

[0179] The embodiment of the present application uses multiple groups of IMU data obtained by continuous detection by the IMU between the first moment and the current moment, the speed of the terminal device obtained by the speed measurement unit at the current moment, and the parallax data to judge whether the terminal device is in a stationary state between the first moment and the current moment, and then determines the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment, and displays the third image through the AR navigation interface, refines the judgment conditions of the stationary state, and can more accurately distinguish different scenarios when the user uses the AR navigation mode, avoids the terminal device from mistakenly using the zero-speed update scheme for navigation and positioning, and thus increases the robustness of the zero-speed update scheme, meets the needs of users in some special scenarios such as uniform motion, large acceleration fluctuations, and the terminal device is blocked, thereby improving the user experience.

[0180] It should be understood that between the time when the above-mentioned application is started and the time when the above-mentioned AR navigation mode is started, and between the time when the above-mentioned AR navigation mode is exited and the time when the above-mentioned application is exited, multiple sets of IMU data obtained by continuous detection through the IMU between the time when the above-mentioned application is started and the time when the above-mentioned AR navigation mode is started are integrated, and multiple sets of IMU data obtained by continuous detection through the IMU between the time when the above-mentioned AR navigation mode is exited and the time when the above-mentioned application is exited are integrated to obtain the position of the terminal device.

[0181] It should also be understood that, between the time when the AR navigation mode is started and the time when the AR navigation mode is exited, if the terminal device is not in a stationary state, it is determined whether the terminal device satisfies the visual update condition between the first moment and the current moment; if the terminal device satisfies the visual update condition between the first moment and the current moment, the visual update is used to determine the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment, and the third image is displayed through the AR navigation interface; if the terminal device does not meet the visual update condition between the first moment and the current moment, multiple sets of IMU data obtained by continuous detection by the IMU between the time when the AR navigation mode is started and the time when the AR navigation mode is exited are integrated to determine the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment, and the third image is displayed through the AR navigation interface.

[0182] As an optional embodiment, multiple groups of IMU data include multiple first accelerations and multiple first angular velocities, and the parallax data include average parallax and parallax ratio. The above S304 includes: based on multiple first accelerations, multiple first angular velocities, the speed of the terminal device at the current moment, the average parallax and the parallax ratio, judging whether the terminal device is in a stationary state between the first moment and the current moment.

[0183] It should be understood that the first acceleration is the modulus of the three-axis acceleration detected by the IMU in the terminal device coordinate system, and the relationship between the first acceleration and the three-axis acceleration can be expressed by the formula:

[0184]

[0185] Similarly, the first angular velocity is the modulus of the three-axis angular velocity detected by the IMU in the terminal device coordinate system. The relationship between the first angular velocity and the three-axis angular velocity can be expressed by the formula:

[0186]

[0187] It should also be understood that there is a corresponding relationship between the above-mentioned multiple first accelerations and the multiple first angular velocities.

[0188] Exemplarily, the IMU detects once every 1 second, and the IMU detects 5 times between the 1st second and the 5th second to obtain 5 groups of IMU data, wherein the corresponding relationship between the multiple first accelerations and the multiple first angular velocities is specifically shown in Table 1:

[0189] Table 1

[0190]

[0191] It should be understood that acc_1, acc_2, acc_3, acc_4 and acc_5 as shown in Table 1 can be a value greater than or equal to 0, and each value greater than or equal to 0 respectively indicates the modulus of the three-axis acceleration obtained by the IMU in the terminal device coordinate system at different detection times; or, acc_1, acc_2, acc_3, acc_4 and acc_5 can also be vectors, and each element in the vector respectively indicates the acceleration of each axis obtained by the IMU in the terminal device coordinate system.

[0192] Similarly, as shown in Table 1, gyo_1, gyo_2, gyo_3, gyo_4 and gyo_5 can be values ​​greater than or equal to 0, and each value greater than or equal to 0 respectively indicates the modulus of the three-axis angular velocity obtained by the IMU in the terminal device coordinate system at different detection times; or, gyo_1, gyo_2, gyo_3, gyo_4 and gyo_5 can also be vectors, and each element in the vector respectively indicates the angular velocity of each axis detected by the IMU in the terminal device coordinate system.

[0193] The embodiment of the present application uses multiple groups of IMU data obtained by continuous detection by the IMU between the first moment and the current moment, the speed of the terminal device obtained by the speed measurement unit at the current moment, and the parallax data to judge whether the terminal device is in a stationary state between the first moment and the current moment, and then determines the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment, and displays the third image through the AR navigation interface, refines the judgment conditions of the stationary state, and can more accurately distinguish different scenarios when the user uses the AR navigation mode, avoids the terminal device from mistakenly using the zero-speed update scheme for navigation and positioning, and thus increases the robustness of the zero-speed update scheme, meets the needs of users in some special scenarios such as uniform motion, large acceleration fluctuations, and the terminal device is blocked, thereby improving the user experience.

[0194] Wherein, judging whether the terminal device is in a stationary state between the first moment and the current moment based on the multiple first accelerations, the multiple first angular velocities, the speed of the terminal device at the current moment, the average parallax, and the parallax ratio includes the following two possible implementation methods:

[0195] In a first possible implementation, the parallax ratio includes a first parallax ratio and a second parallax ratio. When multiple first accelerations are less than or equal to a fourth threshold, multiple first angular velocities are less than or equal to a fifth threshold, the average parallax is less than or equal to a sixth threshold, the first parallax ratio is less than or equal to a seventh threshold, the second parallax ratio is less than or equal to an eighth threshold, and the speed of the terminal device at the current moment is less than or equal to a ninth threshold, it is determined that the terminal device is stationary between the first moment and the current moment.

[0196] Exemplarily, multiple first accelerations and multiple first angular velocities are first judged; when the multiple first accelerations are all less than or equal to the fourth threshold, and the multiple first angular velocities are all less than or equal to the fifth threshold, the average parallax, the first parallax ratio, and the second parallax ratio are then judged; when the average parallax is less than or equal to the sixth threshold, the first parallax ratio is less than or equal to the seventh threshold, and the second parallax ratio is less than or equal to the eighth threshold, the speed of the terminal device at the current moment is finally judged; when the speed of the terminal device at the current moment is less than or equal to the ninth threshold, it is determined that the terminal device is in a stationary state between the first moment and the current moment.

[0197] It should be understood that when the plurality of first accelerations are not all less than or equal to the fourth threshold and the plurality of first angular velocities are not all less than or equal to the fifth threshold, any one of the following three conditions may be met:

[0198] Condition 1: at least one of the plurality of first accelerations is greater than or equal to the fourth threshold, and the plurality of first angular velocities are all less than or equal to the fifth threshold.

[0199] Condition 2: the plurality of first accelerations are all less than or equal to the fourth threshold, and at least one of the plurality of first angular velocities is greater than or equal to the fifth threshold.

[0200] Condition 3: at least one first acceleration among the plurality of first accelerations is greater than or equal to the fourth threshold, and at least one first angular velocity among the plurality of first angular velocities is greater than or equal to the fifth threshold.

[0201] Among them, the situation where condition 1 is met will be described in detail in the second possible implementation method below. When condition 2 or condition 3 is met, it will continue to determine whether the terminal device meets the visual update condition between the first moment and the current moment.

[0202] It should also be understood that when the conditions that the average parallax is less than or equal to the sixth threshold, the first parallax proportion is less than or equal to the seventh threshold, and the second parallax proportion is less than or equal to the eighth threshold are not met, or when the speed of the terminal device at the current moment is not less than or equal to the ninth threshold is not met, it will continue to be determined whether the terminal device meets the visual update condition between the first moment and the current moment.

[0203] The first parallax ratio is the ratio of the number of parallaxes greater than or equal to the second threshold value in the number of the multiple parallaxes, and the second parallax ratio is the ratio of the number of parallaxes greater than or equal to the third threshold value in the number of the multiple parallaxes. The first parallax ratio and the second parallax ratio may be equal or different, and this application does not limit this.

[0204] For example, the above multiple disparities are 2 pixels (pixel, px), 4px, and 8px, respectively, and the disparity greater than or equal to 6px accounts for

[0205] In this way, the embodiments of the present application can prevent the terminal device from mistakenly using the zero-speed update scheme for navigation and positioning in special scenarios where the user is performing uniform motion by limiting the average parallax and the parallax ratio. At the same time, by limiting the speed of the terminal device at the current moment, it prevents the terminal device from using the zero-speed update scheme for navigation and positioning when it has a higher speed, thereby increasing the robustness of the zero-speed update scheme, meeting the needs of users in special scenarios where the user is performing uniform motion, and thus improving the user experience.

[0206] In a second possible implementation, the disparity ratio includes a first disparity ratio and a second disparity ratio, and the disparity data also includes an average feature matching number. When the following conditions are met, it is determined that the terminal device is in a stationary state between the first moment and the current moment: at least one first acceleration among multiple first accelerations is greater than or equal to the fourth threshold, multiple first angular velocities are all less than or equal to the fifth threshold, the average feature matching number is greater than or equal to the tenth threshold, the average disparity is less than or equal to the eleventh threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold; or, at least one first acceleration among multiple first accelerations is greater than or equal to the fourth threshold, multiple first angular velocities are all less than or equal to the fifth threshold, the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold, the average disparity is less than or equal to the thirteenth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

[0207] Exemplarily, multiple first accelerations and multiple first angular velocities are first judged. When there is at least one first acceleration greater than or equal to the fourth threshold value among the multiple first accelerations, and the multiple first angular velocities are all less than or equal to the fifth threshold value, the average feature matching number and the average disparity are then judged. When the average feature matching number is greater than or equal to the tenth threshold value, and the average disparity is less than or equal to the eleventh threshold value, the speed of the terminal device at the current moment is finally judged. When the speed of the terminal device at the current moment is less than or equal to the ninth threshold value, it is determined that the terminal device is in a stationary state between the first moment and the current moment.

[0208] It should be understood that when at least one of the multiple first accelerations is greater than or equal to the fourth threshold and the multiple first angular velocities are all less than or equal to the fifth threshold, any one of the following three conditions may be met:

[0209] Condition 1: the plurality of first accelerations are all less than or equal to the fourth threshold, and the plurality of first angular velocities are all less than or equal to the fifth threshold.

[0210] Condition 2: at least one first acceleration among the plurality of first accelerations is greater than or equal to the fourth threshold, and at least one first angular velocity among the plurality of first angular velocities is greater than or equal to the fifth threshold.

[0211] Condition 3: the plurality of first accelerations are all less than or equal to the fourth threshold, and at least one first angular velocity among the plurality of first angular velocities is greater than or equal to the fifth threshold.

[0212] Among them, the situation where condition 1 is met has been described in detail in the first possible implementation method mentioned above. When condition 2 or condition 3 is met, it will continue to be determined whether the terminal device meets the visual update condition between the first moment and the current moment.

[0213] It should also be understood that when the conditions that the average feature matching number is greater than or equal to the tenth threshold and the average disparity is less than or equal to the eleventh threshold are not satisfied, it will continue to be judged whether the conditions that the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold and the average disparity is less than or equal to the thirteenth threshold are satisfied. When the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold and the average disparity is less than or equal to the thirteenth threshold, it will be judged whether the speed of the terminal device at the current moment is less than or equal to the ninth threshold. When the conditions that the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold and the average disparity is less than or equal to the thirteenth threshold are not satisfied, or when the speed of the terminal device at the current moment is less than or equal to the ninth threshold is not satisfied, it will continue to be judged whether the terminal device meets the visual update condition between the first moment and the current moment.

[0214] It should be understood that the eleventh threshold value and the sixth threshold value may be equal or different, and this application does not limit this. The thirteenth threshold value and the sixth threshold value may be equal or different, and this application does not limit this. The twelfth threshold value should be greater than or equal to the tenth threshold value.

[0215] In this way, the embodiment of the present application can prevent the terminal device from mistakenly using the zero-speed update scheme for navigation positioning in special scenarios where the user is performing a small-range slow rotation by limiting the first angular velocity, thereby increasing the robustness of the zero-speed update scheme, meeting the needs of users in special scenarios of uniform motion, and thus improving the user experience.

[0216] The above two possible implementation methods both use the IMU data obtained by IMU detection to directly determine whether the terminal device is in a stationary state between the first moment and the current moment, and there is a problem that the judgment result is not accurate enough due to the IMU detection error. Therefore, multiple groups of IMU data can be optimized to obtain multiple second accelerations and multiple second angular velocities, and multiple second accelerations, multiple second angular velocities, the speed of the terminal device at the current moment, the average parallax, and the parallax ratio are used to determine whether the terminal device is in a stationary state between the first moment and the current moment.

[0217] The embodiment of the present application uses multiple second accelerations, multiple second angular velocities, the speed of the terminal device at the current moment, the average parallax and the parallax ratio to determine whether the terminal device is in a stationary state between the first moment and the current moment. The specific method is as follows.

[0218] As an optional embodiment, the multiple groups of IMU data include multiple first accelerations and multiple first angular velocities, the parallax data include average parallax and parallax ratio, the above method 300 also includes: based on the multiple first accelerations and the multiple first angular velocities, obtaining multiple second accelerations and multiple second angular velocities, the above S304 also includes: based on the multiple second accelerations, multiple second angular velocities, the speed of the terminal device at the current moment, the average parallax and the parallax ratio, judging whether the terminal device is in a stationary state between the first moment and the current moment.

[0219] It should be understood that there is also a corresponding relationship between the multiple second accelerations and the multiple second angular velocities, which is similar to the corresponding relationship between the multiple first accelerations and the multiple first angular velocities, and will not be elaborated here.

[0220] Wherein, judging whether the terminal device is in a stationary state between the first moment and the current moment based on the multiple second accelerations, the multiple second angular velocities, the speed of the terminal device at the current moment, the average parallax, and the parallax ratio includes the following two possible implementation methods:

[0221] In a first possible implementation, the parallax ratio includes a first parallax ratio and a second parallax ratio. When multiple second accelerations are all less than or equal to a fourth threshold, multiple second angular velocities are all less than or equal to a fifth threshold, the average parallax is less than or equal to a sixth threshold, the first parallax ratio is less than or equal to a seventh threshold, the second parallax ratio is less than or equal to an eighth threshold, and the speed of the terminal device at the current moment is less than or equal to a ninth threshold, it is determined that the terminal device is in a stationary state between the first moment and the current moment.

[0222] It should be understood that the details in this implementation method can refer to the description in the first possible implementation method in which whether the terminal device is in a stationary state between the first moment and the current moment is determined based on multiple first accelerations, multiple first angular velocities, the speed of the terminal device at the current moment, the average parallax and the parallax ratio, and will not be repeated here.

[0223] In a second possible implementation, the disparity ratio includes a first disparity ratio and a second disparity ratio, and the disparity data also includes an average feature matching number. When the following conditions are met, it is determined that the terminal device is in a stationary state between the first moment and the current moment: at least one first acceleration among the multiple second accelerations is greater than or equal to the fourth threshold, the multiple second angular velocities are all less than or equal to the fifth threshold, the average feature matching number is greater than or equal to the tenth threshold, the average disparity is less than or equal to the eleventh threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold; or, at least one first acceleration among the multiple first accelerations is greater than or equal to the fourth threshold, the multiple first angular velocities are all less than or equal to the fifth threshold, the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold, the average disparity is less than or equal to the thirteenth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

[0224] It should be understood that the details in this implementation method can refer to the description in the second possible implementation method of determining whether the terminal device is in a stationary state between the first moment and the current moment based on multiple first accelerations, multiple first angular velocities, the speed of the terminal device at the current moment, the average parallax and the parallax ratio, and will not be repeated here.

[0225] In this way, the embodiment of the present application can obtain multiple second accelerations and multiple second angular velocities by optimizing multiple groups of IMU data respectively, and use multiple second accelerations, multiple second angular velocities, the speed of the terminal device at the current moment, the average parallax and the parallax ratio to determine whether the terminal device is in a stationary state between the first moment and the current moment, thereby compensating for the measurement error of the IMU and increasing the accuracy of the judgment result.

[0226] The specific method of optimizing multiple groups of IMU data to obtain multiple second accelerations and multiple second angular velocities in the embodiment of the present application is as follows.

[0227] As an optional embodiment, the multiple groups of IMU data include multiple first accelerations, multiple first angular velocities and multiple gravitational accelerations. The above method 300 also includes: calculating the absolute value of each first acceleration in the multiple first accelerations and the gravitational acceleration corresponding to each first acceleration to obtain multiple absolute values, and determining the multiple absolute values ​​as multiple second accelerations; determining the multiple first angular velocities as multiple second angular velocities.

[0228] The relationship between the second acceleration and the first acceleration can be expressed by the formula:

[0229] Second acceleration = |first acceleration - acceleration due to gravity|

[0230] For example, the first acceleration is 11 meters per second squared (m / s 2 )、14m / s 2 、15m / s 2 , and the corresponding gravitational acceleration is 9.8m / s 2 , 9.9m / s 2 、9.8m / s 2 , so the second acceleration is 1.2m / s 2 , 4.1m / s 2 , 5.2m / s 2 .

[0231] The relationship between the second angular velocity and the first angular velocity can be expressed by the formula:

[0232] Second angular velocity = first angular velocity

[0233] Exemplarily, the first angular velocities are 0.3 revolutions per second (rad / s), 0.1 rad / s, and 0.4 rad / s, respectively, so the second angular velocities are 0.3 rad / s, 0.1 rad / s, and 0.4 rad / s, respectively.

[0234] It should be understood that there is a corresponding relationship between the above-mentioned multiple first accelerations and the multiple gravitational accelerations.

[0235] Exemplarily, the IMU detects once every 1 second, and the IMU detects 5 times between the 1st second and the 5th second to obtain 5 groups of IMU data, wherein the corresponding relationship between the multiple first accelerations and the multiple gravity accelerations is specifically shown in Table 2:

[0236] Table 2

[0237]

[0238] In this way, the embodiment of the present application can reduce the judgment result error caused by the inaccurate measurement of the acceleration sensor in the IMU, and at the same time, it can also avoid the terminal device from being inaccurate when the gravity acceleration is not 9.81m / s 2 In special scenarios where the zero-speed update solution cannot be used for navigation and positioning, the robustness of the zero-speed update solution is increased, meeting the needs of users in special scenarios of uniform motion, thereby improving the user experience.

[0239] As an optional embodiment, the above S306 includes: if the terminal device is in a stationary state between the first moment and the current moment, determining the update parameters based on whether the terminal device has moved between the start moment of the AR navigation application and the current moment and the number of multiple first images; constructing an equation based on the update parameters to obtain the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment.

[0240] It should be understood that whether the terminal device moves between the startup time of the above-mentioned AR navigation application and the current time can also be understood as the initialization state of the AR navigation application. Specifically, the initialization state of the AR navigation application corresponding to the movement of the terminal device is dynamic initialization, and the initialization state of the AR navigation application corresponding to the non-movement of the terminal device is static initialization.

[0241] Optionally, constructing an equation based on the updated parameters to obtain the position of the terminal device at the current moment and a third image corresponding to the position of the terminal device at the current moment, including: solving the equation using a filtering algorithm to obtain the position of the terminal device at the current moment and a third image corresponding to the position of the terminal device at the current moment.

[0242] It should be understood that the filtering algorithm includes many types, and the embodiments of the present application do not limit the specific filtering algorithm used. Exemplarily, the filtering algorithm may be an extended Kalman filter (EKF) algorithm.

[0243] The embodiment of the present application includes the following two possible implementation methods for determining the update parameters:

[0244] In a first possible implementation method, when the terminal device has not moved between the startup moment and the current moment and the number of multiple first images is greater than or equal to a fourteenth threshold, the speed, displacement and angle between two adjacent first images in the acquisition sequence, and the speed, displacement and angle between the second image and the first image acquired at the first moment are determined as update parameters.

[0245] In a second possible implementation, when the terminal device moves between the startup moment and the current moment, or when the number of multiple first images is less than or equal to a fourteenth threshold, the speed between two adjacent first images in the acquisition sequence and the speed between the second image and the first image acquired at the first moment are determined as update parameters.

[0246] It should be understood that the situation in which the terminal device moves between the above-mentioned starting moment and the current moment, or the number of the plurality of first images is less than or equal to the fourteenth threshold, specifically includes the following three situations:

[0247] Case 1: The terminal device moves between the start time and the current time, and the number of the multiple first images is less than or equal to the fourteenth threshold.

[0248] Case 2: The terminal device moves between the start time and the current time, and the number of the multiple first images is greater than or equal to the fourteenth threshold.

[0249] Case 3: The terminal device does not move between the start time and the current time, and the number of the plurality of first images is less than or equal to the fourteenth threshold.

[0250] In this way, the embodiment of the present application can determine different update parameters according to different scenarios, and then determine the location of the terminal device, thereby improving the accuracy of navigation positioning and the robustness of the zero-speed update solution, thereby improving the user experience.

[0251] As an optional embodiment, the disparity data includes: the average feature matching number, the average disparity, the first disparity ratio and the second disparity ratio. The above S303 includes: when the number of groups of IMU data is greater than or equal to the first threshold, based on the multiple first images and the second images, obtaining multiple feature matching numbers and multiple disparities; determining the average value of the multiple feature matching numbers as the average feature matching number; determining the average value of the multiple disparities as the average disparity; determining the ratio of the number of disparities in the multiple disparities that are greater than or equal to the second threshold to the number of the multiple disparities as the first disparity ratio; determining the ratio of the number of disparities in the multiple disparities that are greater than or equal to the third threshold to the number of the multiple disparities as the second disparity ratio.

[0252] Exemplarily, the plurality of feature matching numbers are 4, 5, and 6, respectively, so the average feature matching number is 5; the plurality of disparities are 3px, 6px, and 3px, respectively, so the average disparity is 4px.

[0253] Optionally, the above-mentioned obtaining multiple feature matching numbers and multiple disparities based on multiple first images and second images includes: using a feature extraction algorithm to extract features from the multiple first images and the second images respectively, to obtain multiple features of each first image in the multiple first images and multiple features of the second image; matching multiple features of two first images adjacent in the acquisition sequence respectively, and matching multiple features of the second image with multiple features of the first image acquired at the first moment respectively, to obtain multiple groups of matching features; determining the number of matching features in each group of the multiple groups of matching features as multiple feature matching numbers; and determining the average matching distance of multiple matching features in each group of matching features as the multiple disparities.

[0254] Exemplarily, three first images (image A, image B and image C) and the second image are sorted from front to back in the order of acquisition, namely: second image, image A, image B, image C, and the feature extraction algorithm is used to obtain multiple features of the second image: a, b, c; the multiple features of image A are: a, b, d, e; the multiple features of image B are: a, c, e, f; the multiple features of image C are e, f, and the multiple features of the second image are matched with the multiple features of image A to obtain a first group of matching features: a, b, and the corresponding feature matching number is 2; the multiple features of image A are matched with the multiple features of image B to obtain a second group of matching features: a, e, and the corresponding feature matching number is 2; the multiple features of image B are matched with the multiple features of image C to obtain a third group of matching features: e, f, and the corresponding feature matching number is 2.

[0255] It should be understood that there are many types of feature extraction algorithms, and the embodiments of the present application do not limit the specific feature extraction algorithm used. Exemplarily, the feature extraction algorithm may be a sparse optical flow (Kanade-Lucas-Tomasi, KTL) algorithm.

[0256] In this way, the embodiment of the present application determines whether the terminal device is in a stationary state between the first moment and the current moment by using the second image corresponding to the position of the terminal device at the first moment and the disparity data between multiple first images continuously collected by the camera between the first moment and the current moment. This can avoid the terminal device from mistakenly using the zero-speed update scheme for navigation and positioning in special scenarios where the user is moving or the waves are peaking and troughing, thereby increasing the robustness of the zero-speed update scheme, meeting the needs of users in special scenarios where they are moving, and thus improving the user experience.

[0257] Below, the navigation and positioning method of the embodiment of the present application is described in detail in combination with the interaction process between the hardware structures such as the IMU, the speed measurement unit and the camera provided on the above-mentioned terminal device.

[0258] Figure 4 The schematic flow chart of the navigation positioning method 400 provided in the embodiment of the present application is shown. The method 400 can be applied to a terminal device supporting the AR navigation mode, and the terminal device is provided with an IMU, a speed measurement unit and a camera. The hardware structure of the terminal device can be as follows Figure 1 As shown, the software structure of the terminal device can be as follows Figure 2 As shown, but the embodiments of the present application are not limited thereto. Figure 4 As shown, the method 400 may include the following steps:

[0259] S401, the IMU sends multiple sets of IMU data to the processor. Correspondingly, the processor receives the multiple sets of IMU data.

[0260] S402: The camera sends a plurality of first images to the processor. Correspondingly, the processor receives the plurality of first images.

[0261] S403: The speed measuring unit sends the speed of the terminal device to the processor. Correspondingly, the processor receives the speed of the terminal device.

[0262] It should be understood that the above S401 to S403 are executed after the above application is started, and the above S401 to S403 are not limited in sequence.

[0263] During the time between the AR navigation mode startup moment and the AR navigation mode exit moment, taking the time between the first moment and the current moment as an example, the method 400 may further include S404 to S406:

[0264] S404: When the number of the multiple groups of IMU data is greater than or equal to the first threshold, the processor obtains disparity data based on the multiple first images and the second image corresponding to the position of the terminal device at the first moment.

[0265] It should be understood that the multiple groups of IMU data in the above S404 are continuously detected by the IMU between the first moment and the current moment, the multiple first images in the above S404 are continuously collected by the camera between the first moment and the current moment, and the speed of the terminal device in the above S404 is the speed of the terminal device at the current moment.

[0266] S405. The processor uses multiple sets of IMU data, the speed and parallax data of the terminal device at the current moment to determine whether the terminal device is in a stationary state between the first moment and the current moment.

[0267] S406. If the terminal device is in a stationary state between the first moment and the current moment, the processor determines the position of the terminal device at the current moment and a third image corresponding to the position of the terminal device at the current moment, and displays the third image through the AR navigation interface.

[0268] It should be understood that between the time when the above-mentioned application is started and the time when the above-mentioned AR navigation mode is started, and between the time when the above-mentioned AR navigation mode is exited and the time when the above-mentioned application is exited, the processor integrates the multiple sets of IMU data received to obtain the position of the terminal device. Between the time when the above-mentioned AR navigation mode is started and the time when the above-mentioned AR navigation mode is exited, if the terminal device is not in a stationary state, the processor determines whether the terminal device meets the visual update condition between the first moment and the current moment; if the terminal device meets the visual update condition between the first moment and the current moment, the processor uses the visual update to determine the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment, and displays the third image through the AR navigation interface; if the terminal device does not meet the visual update condition between the first moment and the current moment, the processor integrates the multiple sets of IMU data received to determine the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment, and displays the third image through the AR navigation interface.

[0269] The embodiment of the present application uses multiple groups of IMU data obtained by continuous detection by the IMU between the first moment and the current moment, the speed of the terminal device obtained by the speed measurement unit at the current moment, and the parallax data to judge whether the terminal device is in a stationary state between the first moment and the current moment, and then determines the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment, and displays the third image through the AR navigation interface, refines the judgment conditions of the stationary state, and can more accurately distinguish different scenarios when the user uses the AR navigation mode, avoids the terminal device from mistakenly using the zero-speed update scheme for navigation and positioning, and thus increases the robustness of the zero-speed update scheme, meets the needs of users in some special scenarios such as uniform motion, large acceleration fluctuations, and the terminal device is blocked, thereby improving the user experience.

[0270] in, Figure 4 The method 400 shown is Figure 3 The method 300 shown has no substantial difference and is merely for ease of understanding, so no further elaboration is given here.

[0271] Next, combine Figure 5 and Figure 6 The navigation and positioning method of the embodiment of the present application is described in detail.

[0272] Figure 5 The schematic flow chart of the navigation positioning method 500 provided in the embodiment of the present application is shown. The method 500 can be applied to a terminal device supporting the AR navigation mode, and the terminal device is provided with an IMU, a speed measurement unit and a camera. The hardware structure of the terminal device can be as follows Figure 1 As shown, the software structure of the terminal device can be as follows Figure 2 As shown, but the embodiments of the present application are not limited thereto. Figure 5 As shown, the method 500 may include the following steps:

[0273] S501. After the user starts an application for navigation and positioning through a terminal device, multiple groups of IMU data are obtained through detection through the IMU, multiple first images are obtained through acquisition through the camera, and the speed of the terminal device is obtained through detection through the speed measurement unit.

[0274] S502: The terminal device integrates multiple sets of IMU data to obtain the position of the terminal device.

[0275] After the user starts the AR navigation mode through the terminal device, the above S502 can be replaced by S503:

[0276] S503. The terminal device determines whether the following conditions are met: the number of multiple groups of IMU data obtained by the continuous detection of the IMU between the first moment and the current moment is greater than or equal to the first threshold.

[0277] If the judgment result obtained by the terminal device in S503 is that the condition is met, the terminal device continues to execute S504 after the above S503, specifically:

[0278] S504: The terminal device determines whether a visual update condition is met.

[0279] If the terminal device obtains the judgment result in S504 that the visual update condition is met, the terminal device continues to execute S505 after S504, specifically:

[0280] S505: The terminal device obtains the location of the terminal device by using visual update.

[0281] If the terminal device obtains the judgment result in S504 that the visual update condition is not met, the terminal device directly returns to execute S502 after the above S504.

[0282] After the above S502, the method 500 further includes S506 and S507:

[0283] S506. The terminal device obtains disparity data based on a plurality of first images continuously acquired by the camera between the first moment and the current moment and a second image corresponding to the position of the terminal device at the first moment.

[0284] S507: The terminal device obtains an average disparity, a first disparity ratio, a second disparity ratio, and an average feature matching number based on the disparity data.

[0285] In the above S503, if the judgment result obtained by the terminal device is that the condition is not met, the terminal device may directly execute S509 after the above S503. Optionally, the terminal device may also directly execute S508 after the above S503. Specifically:

[0286] S508, the terminal device optimizes multiple accelerations and multiple angular velocities in multiple sets of IMU data continuously detected by the IMU between the first moment and the current moment. Optionally, the terminal device calculates the absolute value of each acceleration in the multiple sets of IMU data and the gravity acceleration corresponding to each acceleration to obtain multiple absolute values, and determines the multiple absolute values ​​as multiple optimized accelerations; and determines each angular velocity in the multiple sets of IMU data as multiple optimized angular velocities.

[0287] S509. The terminal device determines whether the following conditions are met: multiple accelerations are all less than or equal to a fourth threshold, and multiple angular velocities are all less than or equal to a fifth threshold.

[0288] It should be understood that if the terminal device directly executes S509 after the above S503, the multiple accelerations and multiple angular velocities in S509 are respectively the multiple accelerations in the multiple groups of IMU data and the multiple angular velocities in the multiple groups of IMU data; if the terminal device directly executes S508 after the above S503, the multiple accelerations and multiple angular velocities in S509 are respectively the multiple accelerations optimized in S508 and the multiple angular velocities optimized in S508.

[0289] If the judgment result obtained by the terminal device in S509 is that the condition is met, the terminal device continues to execute S510 after the above S509, specifically:

[0290] S510: The terminal device determines whether the following conditions are met: the average disparity is less than or equal to a sixth threshold, the first disparity ratio is less than or equal to a seventh threshold, and the second disparity ratio is less than or equal to an eighth threshold.

[0291] If the judgment result obtained by the terminal device in S510 is that the condition is met, the terminal device continues to execute S511 after the above S510, specifically:

[0292] S511. The terminal device determines whether the following conditions are met: the speed of the terminal device at the current moment is less than or equal to a ninth threshold.

[0293] If the judgment result obtained by the terminal device in S511 is that the condition is met, the terminal device continues to execute S512 after the above S511, specifically:

[0294] S512. The terminal device determines whether the following conditions are met: the terminal device has not moved between the startup time and the current time, and the number of multiple first images continuously acquired by the camera between the first time and the current time is greater than or equal to the fourteenth threshold.

[0295] If the judgment result obtained by the terminal device in S512 is that the condition is met, the terminal device continues to execute S513 after the above S512, specifically:

[0296] S513, the terminal device determines the speed, displacement and angle between two first images adjacent in the acquisition order, and the speed, displacement and angle between the second image and the first image acquired by the camera at the first moment as update parameters, constructs an equation based on the update parameters, and obtains the position of the terminal device.

[0297] If the terminal device obtains the judgment result in S512 that the condition is not met, the terminal device directly executes S514 after S512, specifically:

[0298] S514: The terminal device determines the speed between two first images that are adjacent in acquisition order, and the speed between the second image and the first image acquired by the camera at the first moment as update parameters, constructs an equation based on the update parameters, and obtains the position of the terminal device.

[0299] In the above S509, if the terminal device obtains the judgment result that the condition is not met, the terminal device directly executes S515 after the above S509, specifically:

[0300] S515. The terminal device determines whether the following conditions are met: the average feature matching number is greater than or equal to the tenth threshold, and the average disparity is less than or equal to the eleventh threshold, or the average feature matching number is less than or equal to the twelfth threshold and greater than or equal to the tenth threshold, and the average disparity is less than or equal to the thirteenth threshold.

[0301] If the judgment result obtained by the terminal device in S515 is that the condition is met, the terminal device continues to execute S516 after the above S515, specifically:

[0302] S516. The terminal device determines whether the following conditions are met: multiple angular velocities are all less than or equal to the fifth threshold.

[0303] If the judgment result obtained by the terminal device in S516 satisfies the condition, the terminal device returns to execute S511 after the above S516.

[0304] If the judgment result obtained by the terminal device in S516 is that the condition is not met, the terminal device returns to execute S504 after the above S516.

[0305] In the above S510, if the judgment result obtained by the terminal device is that the condition is not met, the terminal device returns to execute S504 after the above S510.

[0306] In the above S511, if the judgment result obtained by the terminal device is that the condition is not met, the terminal device returns to execute S504 after the above S511.

[0307] In the above S515, if the judgment result obtained by the terminal device is that the condition is not met, the terminal device returns to execute S504 after the above S515.

[0308] Figure 6 The schematic flow chart of the navigation positioning method 600 provided in the embodiment of the present application is shown. The method 600 can be applied to a terminal device supporting the AR navigation mode, and the terminal device is provided with an IMU, a speed measurement unit and a camera. The hardware structure of the terminal device can be as follows Figure 1 As shown, the software structure of the terminal device can be as follows Figure 2 As shown, but the embodiments of the present application are not limited thereto. Figure 6 As shown, the method 600 may include the following steps:

[0309] S601. After the user starts an application for navigation and positioning through a terminal device, multiple groups of IMU data are obtained through detection through the IMU, multiple first images are obtained through acquisition through the camera, and the speed of the terminal device is obtained through detection through the speed measurement unit.

[0310] S602: The terminal device integrates multiple sets of IMU data to obtain the position of the terminal device.

[0311] After the user starts the AR navigation mode through the terminal device, the above S602 can be replaced by S603:

[0312] S603. The terminal device determines whether the following conditions are met: whether the number of multiple groups of IMU data obtained by the IMU through continuous detection between the first moment and the current moment is greater than or equal to a first threshold.

[0313] S604: The terminal device determines whether a visual update condition is met.

[0314] S605: The terminal device obtains the location of the terminal device by using visual update.

[0315] After the above S602, the method 500 further includes S606 and S607:

[0316] S606. The terminal device obtains disparity data based on a plurality of first images continuously acquired by the camera between the first moment and the current moment and a second image corresponding to the position of the terminal device at the first moment.

[0317] S607: The terminal device obtains an average disparity, a first disparity ratio, a second disparity ratio, and an average feature matching number based on the disparity data.

[0318] In the above S603, if the judgment result obtained by the terminal device is that the condition is not met, the terminal device may directly execute S609 after the above S603. Optionally, the terminal device may also directly execute S608 after the above S603. Specifically:

[0319] S608, the terminal device optimizes multiple accelerations and multiple angular velocities in multiple sets of IMU data continuously detected by the IMU between the first moment and the current moment. Optionally, the terminal device calculates the absolute value of each acceleration in the multiple sets of IMU data and the gravity acceleration corresponding to each acceleration to obtain multiple absolute values, and determines the multiple absolute values ​​as multiple optimized accelerations; and determines each angular velocity in the multiple sets of IMU data as multiple optimized angular velocities.

[0320] S609. The terminal device determines whether the following conditions are met: multiple angular velocities are all less than or equal to a fifth threshold.

[0321] The above S601 to S608 may refer to the description of the above S501 to S508, which will not be repeated here.

[0322] It should be understood that if the terminal device directly executes S609 after the above S603, the multiple angular velocities in S609 are the multiple angular velocities in multiple groups of IMU data; if the terminal device directly executes S608 after the above S603, the multiple angular velocities in S609 are the multiple angular velocities optimized in S608.

[0323] If the judgment result obtained by the terminal device in S609 is that the condition is met, the terminal device continues to execute S610 after the above S609, specifically:

[0324] S610. The terminal device determines whether the following conditions are met: multiple accelerations are all less than or equal to a fourth threshold.

[0325] It should be understood that if the terminal device directly executes S609 after the above S603, the multiple accelerations in S610 are the multiple accelerations in multiple groups of IMU data; if the terminal device directly executes S608 after the above S603, the multiple accelerations in S610 are the multiple accelerations optimized in S608.

[0326] If the judgment result obtained by the terminal device in S610 is that the condition is met, the terminal device continues to execute S611 after the above S610, specifically:

[0327] S611. The terminal device determines whether the following conditions are met: the average disparity is less than or equal to the sixth threshold, the first disparity ratio is less than or equal to the seventh threshold, and the second disparity ratio is less than or equal to the eighth threshold.

[0328] S612. The terminal device determines whether the following conditions are met: the speed of the terminal device at the current moment is less than or equal to a ninth threshold.

[0329] S613. The terminal device determines whether the following conditions are met: the terminal device has not moved between the start time and the current time, and the number of multiple first images continuously acquired by the camera between the first time and the current time is greater than or equal to the fourteenth threshold.

[0330] S614, the terminal device determines the speed, displacement and angle between two adjacent first images in the acquisition order, and the speed, displacement and angle between the second image and the first image acquired by the camera at the first moment as update parameters, constructs an equation based on the update parameters, and obtains the position of the terminal device.

[0331] S615. The terminal device determines the speed between two first images that are adjacent in acquisition order, and the speed between the second image and the first image acquired by the camera at the first moment as update parameters, constructs an equation based on the update parameters, and obtains the position of the terminal device.

[0332] The above S611 to S615 may refer to the description of the above S510 to S514, which will not be repeated here.

[0333] In the above S609, if the judgment result obtained by the terminal device is that the condition is not met, the terminal device returns to execute S604 after the above S609.

[0334] In the above S610, if the terminal device obtains the judgment result that the condition is not met, the terminal device continues to execute S616 after the above S610, specifically:

[0335] S616. The terminal device determines whether the following conditions are met: the average feature matching number is greater than or equal to the tenth threshold, and the average disparity is less than or equal to the eleventh threshold, or the average feature matching number is less than or equal to the twelfth threshold and greater than or equal to the tenth threshold, and the average disparity is less than or equal to the thirteenth threshold.

[0336] If the judgment result obtained by the terminal device in S616 is that the condition is met, the terminal device directly executes S612 after the above S616.

[0337] If the judgment result obtained by the terminal device in S616 is that the condition is not met, the terminal device returns to execute S604 after the above S616.

[0338] It should be understood that Figure 5 and Figure 6 These are only examples. In addition, the specific order of comparing the multiple groups of IMU data, the speed of the terminal device at the current moment, the disparity data and other data with the corresponding thresholds in method 300 may also include other possible implementation methods, which are not limited to the embodiments of the present application.

[0339] It can be understood that in the embodiments of the present application, the size of the serial numbers of the above-mentioned processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0340] Combined with the above Figures 1 to 6 , describes the method of the embodiment of the present application in detail, and will be combined with Figure 7 , describe in detail the terminal device of an embodiment of the present application.

[0341] Figure 7 A schematic block diagram of a terminal device 700 provided in an embodiment of the present application is shown. The terminal device 700 supports the AR navigation mode, and the terminal device 700 includes an IMU 701, a speed measuring unit 702, a camera 703, a processor 704, and a memory 705. Among them, the IMU 701 is used to continuously detect multiple groups of IMU data, the speed measuring unit 702 is used to detect the speed of the terminal device, the camera 703 is used to continuously collect multiple images, the processor 704 and the memory 705 communicate with each other through an internal connection path, the memory 705 is used to store instructions, and the processor 704 is used to execute the instructions stored in the memory 705.

[0342] Specifically, IMU 701 is used to: continuously perform detection between the first moment and the current moment to obtain multiple groups of IMU data; the speed measurement unit 702 is used to: perform detection at the current moment to obtain the speed of the terminal device at the current moment; the camera 703 is used to: continuously collect between the first moment and the current moment to obtain multiple first images; the processor 704 is used to: when the number of groups of multiple groups of IMU data is greater than or equal to the first threshold, based on the multiple first images and the second image corresponding to the position of the terminal device at the first moment, obtain disparity data; the processor 704 is also used to: use the multiple groups of IMU data, the speed of the terminal device at the current moment and the disparity data to determine whether the terminal device is in a stationary state between the first moment and the current moment; the processor 704 is also used to: if the terminal device is in a stationary state between the first moment and the current moment, determine the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment, and display the third image through the AR navigation interface.

[0343] Optionally, the multiple groups of IMU data include multiple first accelerations and multiple first angular velocities, and the parallax data include average parallax and parallax ratio; the processor 704 is also used to: determine whether the terminal device is in a stationary state between the first moment and the current moment based on the multiple first accelerations, multiple first angular velocities, the speed of the terminal device at the current moment, the average parallax and the parallax ratio.

[0344] Optionally, the parallax ratio includes a first parallax ratio and a second parallax ratio, the first parallax ratio being the ratio of the number of parallaxes greater than or equal to the second threshold in the number of multiple parallaxes, and the second parallax ratio being the ratio of the number of parallaxes greater than or equal to the third threshold in the number of multiple parallaxes; the processor 704 is also used to: determine that the terminal device is in a stationary state between the first moment and the current moment when the following conditions are met: multiple first accelerations are all less than or equal to the fourth threshold, multiple first angular velocities are all less than or equal to the fifth threshold, the average parallax is less than or equal to the sixth threshold, the first parallax ratio is less than or equal to the seventh threshold, the second parallax ratio is less than or equal to the eighth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

[0345] Optionally, the disparity data also includes: an average feature matching number; the processor 704 is also used to: determine that the terminal device is in a stationary state between the first moment and the current moment when the following conditions are met: at least one first acceleration among the multiple first accelerations is greater than or equal to the fourth threshold, the multiple first angular velocities are all less than or equal to the fifth threshold, the average feature matching number is greater than or equal to the tenth threshold, the average disparity is less than or equal to the eleventh threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold; or, at least one first acceleration among the multiple first accelerations is greater than or equal to the fourth threshold, the multiple first angular velocities are all less than or equal to the fifth threshold, the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold, the average disparity is less than or equal to the thirteenth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

[0346] Optionally, the parallax ratio includes a first parallax ratio and a second parallax ratio, the first parallax ratio being the ratio of the number of parallaxes greater than or equal to the second threshold in the number of multiple parallaxes, and the second parallax ratio being the ratio of the number of parallaxes greater than or equal to the third threshold in the number of multiple parallaxes; the processor 704 is also used to: obtain multiple second accelerations and multiple second angular velocities based on multiple first accelerations and multiple first angular velocities; the processing unit 802 is also used to: determine that the terminal device is in a stationary state between the first moment and the current moment when the following conditions are met: multiple second accelerations are all less than or equal to the fourth threshold, multiple second angular velocities are all less than or equal to the fifth threshold, the average parallax is less than or equal to the sixth threshold, the first parallax ratio is less than or equal to the seventh threshold, the second parallax ratio is less than or equal to the eighth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

[0347] Optionally, the disparity data also includes: an average feature matching number; the processor 704 is also used to: determine that the terminal device is in a stationary state between the first moment and the current moment when the following conditions are met: at least one second acceleration among the multiple second accelerations is greater than or equal to the fourth threshold, the multiple second angular velocities are all less than or equal to the fifth threshold, the average feature matching number is greater than or equal to the tenth threshold, the average disparity is less than or equal to the eleventh threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold; or, at least one second acceleration among the multiple second accelerations is greater than or equal to the fourth threshold, the multiple second angular velocities are all less than or equal to the fifth threshold, the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold, the average disparity is less than or equal to the thirteenth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

[0348] Optionally, the multiple groups of IMU data also include multiple gravitational accelerations; the processor 704 is also used to: calculate the absolute value of each first acceleration in the multiple first accelerations and the gravitational acceleration corresponding to each first acceleration to obtain multiple absolute values, and determine the multiple absolute values ​​as multiple second accelerations; determine the multiple first angular velocities as multiple second angular velocities.

[0349] Optionally, the processor 704 is also used to: if the terminal device is in a stationary state between the first moment and the current moment, determine the update parameters based on whether the terminal device has moved between the start moment of the AR navigation application and the current moment and the number of multiple first images; construct an equation based on the update parameters to obtain the position of the terminal device at the current moment and the third image corresponding to the position of the terminal device at the current moment.

[0350] Optionally, the processor 704 is also used to: when the terminal device has not moved between the startup moment and the current moment, and the number of multiple first images is greater than or equal to a fourteenth threshold, determine the speed, displacement and angle between two adjacent first images in the acquisition sequence, and the speed, displacement and angle between the second image and the first image acquired at the first moment as update parameters.

[0351] Optionally, the processor 704 is also used to: when the terminal device moves between the startup moment and the current moment, or when the number of multiple first images is less than or equal to a fourteenth threshold, determine the speed between two adjacent first images in the acquisition sequence, and the speed between the second image and the first image acquired at the first moment as update parameters.

[0352] Optionally, the processor 704 is further configured to: solve the equation using a filtering algorithm to obtain the position of the terminal device at the current moment and a third image corresponding to the position of the terminal device at the current moment.

[0353] Optionally, the processor 704 is also used to: obtain multiple feature matching numbers and multiple disparities based on multiple first images and second images; determine the average value of the multiple feature matching numbers as the average feature matching number; determine the average value of the multiple disparities as the average disparity; determine the ratio of the number of disparities in the multiple disparities that are greater than or equal to the second threshold to the number of multiple disparities as the first disparity ratio; determine the ratio of the number of disparities in the multiple disparities that are greater than or equal to the third threshold to the number of multiple disparities as the second disparity ratio.

[0354] Optionally, the processor 704 is also used to: use a feature extraction algorithm to extract features from multiple first images and second images respectively, to obtain multiple features of each first image in the multiple first images and multiple features of the second image; match multiple features of two first images that are adjacent in acquisition order respectively, and match multiple features of the second image with multiple features of the first image acquired at the first moment respectively, to obtain multiple groups of matching features; determine the number of matching features in each group of the multiple groups of matching features as multiple feature matching numbers; determine the average matching distance of multiple matching features in each group of matching features as multiple disparities.

[0355] It should be understood that the processor 704 and the memory 705 can also communicate with the transceiver through an internal connection path, and the processor 704 is also used to control the transceiver to receive and / or send signals by executing instructions stored in the memory 705.

[0356] It should be understood that the terminal device 700 can be specifically the terminal device in the above embodiment, and can be used to execute the various steps and / or processes in the above method. Optionally, the memory 705 may include a read-only memory and a random access memory, and provide instructions and data to the processor 704. A part of the memory may also include a non-volatile random access memory. For example, the memory 705 may also store information about the device type. The processor 704 may be used to execute the instructions stored in the memory 705, and when the processor 704 executes the instructions stored in the memory, the processor 704 is used to execute the various steps and / or processes of the above method 700. The transceiver may include a transmitter and a receiver, the transmitter may be used to implement the various steps and / or processes corresponding to the above transceiver for performing the sending action, and the receiver may be used to implement the various steps and / or processes corresponding to the above transceiver for performing the receiving action.

[0357] It should be understood that in the embodiments of the present application, the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc.

[0358] In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in a processor or an instruction in the form of software. The steps of the method disclosed in conjunction with the embodiment of the present application can be directly embodied as a hardware processor for execution, or a combination of hardware and software modules in a processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in a memory, and the processor executes the instructions in the memory, and completes the steps of the above method in conjunction with its hardware. To avoid repetition, it is not described in detail here.

[0359] The present application also provides a computer-readable storage medium, which is used to store a computer program, and the computer program is used to implement the method in the above embodiment.

[0360] The present application also provides a computer program product, which includes a computer program (also referred to as code or instruction). When the computer program runs on a computer, the computer can execute the method in the above embodiment.

[0361] Those of ordinary skill in the art will appreciate that the modules and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0362] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and modules described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0363] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0364] The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical modules, that is, they may be located in one place or distributed on multiple network modules. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0365] In addition, each functional module in each embodiment of the present application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0366] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0367] The above is only a specific implementation of the present application, but the protection scope of the embodiments of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the embodiments of the present application, which should be included in the protection scope of the embodiments of the present application. Therefore, the protection scope of the embodiments of the present application should be based on the protection scope of the claims.

Claims

1. A navigation and positioning method, characterized in that: Applied to a terminal device supporting an augmented reality (AR) navigation mode, the terminal device is provided with an inertial measurement unit (IMU), a speed measurement unit, and a camera, and the method comprises: Continuously performing detection by the IMU between the first moment and the current moment to obtain multiple sets of IMU data; The speed measuring unit detects at the current moment to obtain the speed of the terminal device at the current moment; When the number of the multiple groups of IMU data is greater than or equal to a first threshold, obtaining disparity data based on a plurality of first images and a second image corresponding to the position of the terminal device at the first moment, wherein the plurality of first images are continuously collected by the camera between the first moment and the current moment; Using the multiple sets of IMU data, the speed of the terminal device at the current moment, and the parallax data, determine whether the terminal device is in a stationary state between the first moment and the current moment; If the terminal device is stationary between the first moment and the current moment, and the terminal device moves between the start-up moment of the AR navigation application and the current moment, or the number of the plurality of first images is less than or equal to a fourteenth threshold, a speed between two first images adjacent in acquisition order and a speed between the second image and the first image acquired at the first moment are determined as update parameters; Based on the updated parameter construction equation, the position of the terminal device at the current moment and a third image corresponding to the position of the terminal device at the current moment are obtained, and the third image is displayed through the AR navigation interface.

2. The method according to claim 1, characterized in that The multiple groups of IMU data include multiple first accelerations and multiple first angular velocities, and the disparity data include average disparity and disparity ratio; The using the multiple sets of IMU data, the speed of the terminal device at the current moment, and the parallax data to determine whether the terminal device is in a stationary state between the first moment and the current moment includes: Based on the multiple first accelerations, the multiple first angular velocities, the speed of the terminal device at the current moment, the average parallax, and the parallax ratio, it is determined whether the terminal device is in a stationary state between the first moment and the current moment.

3. The method according to claim 2, characterized in that The parallax ratio includes a first parallax ratio and a second parallax ratio, wherein the first parallax ratio is a ratio of the number of parallaxes greater than or equal to a second threshold in the number of the multiple parallaxes, and the second parallax ratio is a ratio of the number of parallaxes greater than or equal to a third threshold in the number of the multiple parallaxes; The determining whether the terminal device is in a stationary state between the first moment and the current moment includes: When the following conditions are met, it is determined that the terminal device is in a stationary state between the first moment and the current moment: The multiple first accelerations are all less than or equal to the fourth threshold, the multiple first angular velocities are all less than or equal to the fifth threshold, the average parallax is less than or equal to the sixth threshold, the first parallax proportion is less than or equal to the seventh threshold, the second parallax proportion is less than or equal to the eighth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

4. The method according to claim 3, characterized in that The disparity data also includes: an average feature matching number; The determining whether the terminal device is in a stationary state between the first moment and the current moment further includes: When the following conditions are met, it is determined that the terminal device is in a stationary state between the first moment and the current moment: At least one of the multiple first accelerations is greater than or equal to the fourth threshold, the multiple first angular velocities are all less than or equal to the fifth threshold, the average feature matching number is greater than or equal to the tenth threshold, the average parallax is less than or equal to the eleventh threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold; or, Among the multiple first accelerations, there is at least one first acceleration greater than or equal to the fourth threshold, the multiple first angular velocities are all less than or equal to the fifth threshold, the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold, the average parallax is less than or equal to the thirteenth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

5. The method according to claim 2, characterized in that: The parallax ratio includes a first parallax ratio and a second parallax ratio, wherein the first parallax ratio is a ratio of the number of parallaxes greater than or equal to a second threshold in the number of the multiple parallaxes, and the second parallax ratio is a ratio of the number of parallaxes greater than or equal to a third threshold in the number of the multiple parallaxes; The method further comprises: Based on the plurality of first accelerations and the plurality of first angular velocities, obtaining a plurality of second accelerations and a plurality of second angular velocities; The determining whether the terminal device is in a stationary state between the first moment and the current moment includes: When the following conditions are met, it is determined that the terminal device is in a stationary state between the first moment and the current moment: The multiple second accelerations are all less than or equal to the fourth threshold, the multiple second angular velocities are all less than or equal to the fifth threshold, the average parallax is less than or equal to the sixth threshold, the first parallax proportion is less than or equal to the seventh threshold, the second parallax proportion is less than or equal to the eighth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

6. The method according to claim 5, characterized in that The disparity data also includes: an average feature matching number; The determining whether the terminal device is in a stationary state between the first moment and the current moment further includes: When the following conditions are met, it is determined that the terminal device is in a stationary state between the first moment and the current moment: At least one of the multiple second accelerations is greater than or equal to the fourth threshold, the multiple second angular velocities are all less than or equal to the fifth threshold, the average feature matching number is greater than or equal to the tenth threshold, the average parallax is less than or equal to the eleventh threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold; or, Among the multiple second accelerations, there is at least one second acceleration greater than or equal to the fourth threshold, the multiple second angular velocities are all less than or equal to the fifth threshold, the average feature matching number is less than or equal to the tenth threshold and greater than or equal to the twelfth threshold, the average parallax is less than or equal to the thirteenth threshold, and the speed of the terminal device at the current moment is less than or equal to the ninth threshold.

7. The method according to claim 5 or 6, characterized in that: The multiple sets of IMU data also include multiple gravitational accelerations; The obtaining a plurality of second accelerations and a plurality of second angular velocities based on the plurality of first accelerations and the plurality of first angular velocities comprises: Calculating an absolute value of each first acceleration in the plurality of first accelerations and a gravity acceleration corresponding to each first acceleration as a difference to obtain a plurality of absolute values, and determining the plurality of absolute values ​​as the plurality of second accelerations; The plurality of first angular velocities are determined as the plurality of second angular velocities.

8. The method according to claim 1, characterized in that The method further comprises: If the terminal device is in a stationary state between the first moment and the current moment, and the terminal device has not moved between the start moment and the current moment and the number of the multiple first images is greater than or equal to a fourteenth threshold, the speed, displacement and angle between two adjacent first images in the acquisition sequence, and the speed, displacement and angle between the second image and the first image acquired at the first moment are determined as the update parameters.

9. The method according to claim 1 or 8, characterized in that: The constructing an equation based on the updated parameters to obtain the position of the terminal device at the current moment and a third image corresponding to the position of the terminal device at the current moment includes: The equation is solved by using a filtering algorithm to obtain the position of the terminal device at the current moment and a third image corresponding to the position of the terminal device at the current moment.

10. The method according to any one of claims 1 to 5 and 8, characterized in that The obtaining disparity data based on the plurality of first images and the second image corresponding to the position of the terminal device at the first moment includes: Based on the plurality of first images and the second image, obtaining a plurality of feature matching numbers and a plurality of disparities; Determine an average value of the plurality of feature matching numbers as the average feature matching number; determining an average value of the plurality of disparities as the average disparity; determining a ratio of the number of disparities greater than or equal to the second threshold in the number of the disparities as the first disparity ratio; The ratio of the number of disparities greater than or equal to the third threshold in the number of the disparities is determined as the second disparity ratio.

11. The method according to claim 10, characterized in that The obtaining a plurality of feature matching numbers and a plurality of disparities based on the plurality of first images and the second images comprises: Using a feature extraction algorithm, extract features from the plurality of first images and the second image respectively to obtain a plurality of features of each of the plurality of first images and a plurality of features of the second image; Matching multiple features of two first images that are adjacent in acquisition order respectively, and matching multiple features of the second image with multiple features of the first image acquired at the first moment respectively, to obtain multiple groups of matching features; Determining the number of matching features in each group of the multiple groups of matching features as the multiple feature matching numbers; The average matching distances of the multiple matching features in each group of matching features are determined as the multiple disparities.

12. A terminal device, characterized in that: include: Inertial measurement unit IMU, speed measurement unit, camera and processor; The processor is coupled to a memory, and the memory is used to store a computer program. When the processor calls the computer program, the terminal device executes the method as described in any one of claims 1 to 11 through the IMU, the speed measurement unit, and the camera.

13. A computer-readable storage medium, characterized in that: Used to store a computer program, the computer program comprising instructions for implementing the method according to any one of claims 1 to 11.

14. A computer program product, characterized in that The computer program product includes a computer program code, and when the computer program code is executed on a computer, the computer is enabled to implement the method according to any one of claims 1 to 11.

Images