Navigation positioning method and device and storage medium

Abstract

The invention relates to a navigation positioning method and device and a storage medium. The navigation positioning method comprises the following steps: acquiring continuous multi-frame images of a video shot by a terminal camera; according to the continuous multiple frames of images, the course variation is determined in real time, and the course variation represents the variation of the movement direction of the terminal; and determining the current movement direction of the terminal based on the course variation and the historical movement direction of the terminal when the camera starts to shoot the continuous multiple frames of images. According to the invention, the moving direction of the pedestrian can be accurately determined.

Description

Technical Field

[0001] The present disclosure relates to the field of positioning, and in particular to a navigation positioning method, device and storage medium. Background Art

[0002] With the development of satellite technology and chip technology, the public is increasingly dependent on the positioning and navigation of mobile terminals such as mobile phones, and the requirements for the positioning accuracy of mobile phones are also getting higher and higher. Combined navigation technology generally uses pedestrian movement status, map and other data to fuse with the Global Navigation Satellite System (GNSS) data to optimize inertial navigation to meet the public's application needs in scenes such as occluded environments and multipath environments. However, in the process of inertial navigation optimization, the threshold is set using the gait characteristics of pedestrians while walking to capture quasi-static points. This method is very prone to errors, resulting in negative optimization of positioning, thereby affecting the positioning effect. Summary of the invention

[0003] In order to overcome the problems existing in the related art, the present disclosure provides a navigation and positioning method, device and storage medium.

[0004] According to the first aspect of an embodiment of the present disclosure, a navigation and positioning method is provided, comprising: acquiring a plurality of continuous frames of images of a video captured by a terminal camera; determining a heading change in real time based on the plurality of continuous frames of images, the heading change representing a change in the direction of movement of the terminal; and determining the current direction of movement of the terminal based on the heading change and the historical direction of movement of the terminal when the camera starts capturing the plurality of continuous frames of images.

[0005] In one embodiment, before acquiring a plurality of continuous frames of video captured by a terminal camera, the method further includes: determining that a motion state data value of the terminal is less than a threshold value, wherein the motion state data includes at least acceleration and angular acceleration.

[0006] In one embodiment, determining the heading change amount in real time based on the continuous multi-frame images includes: marking feature points of the same object in the continuous multi-frame images, where there are one or more corresponding feature points in the images of adjacent frames in the continuous multi-frame images, and the corresponding feature points represent the relative position of the same object in the images of adjacent frames in the continuous multi-frame images; determining the relative offset of the corresponding feature points in real time based on the relative position of the corresponding feature points in the images of adjacent frames in the continuous multi-frame images; and determining the heading change amount based on the relative offset of the corresponding feature points.

[0007] In one embodiment, determining the heading change amount based on the relative offset of the corresponding feature point includes: determining a first heading change amount in real time through an inertial sensor in the terminal; using the relative offset as a second heading change amount, the relative offset being determined in real time based on the relative position of the corresponding feature point in adjacent frames of the continuous multiple frames of images; and merging the first heading change amount and the second heading change amount according to a preset weight to obtain the heading change amount.

[0008] In one embodiment, the current movement direction of the terminal is determined based on the heading change and the movement direction of the terminal when starting to acquire the continuous multiple frames of images, including: determining the historical movement direction of the terminal when the camera starts to shoot the continuous multiple frames of images; and superimposing the historical movement direction of the terminal and the heading change in real time through Kalman filtering to obtain the current movement direction of the terminal.

[0009] According to the second aspect of an embodiment of the present disclosure, a navigation and positioning device is provided, comprising: a receiving module, used to obtain a plurality of continuous frames of images of a video captured by a terminal camera; a processing module, used to determine a heading change in real time based on the plurality of continuous frames of images, wherein the heading change represents a change in the direction of movement of the terminal; and determine the current direction of movement of the terminal based on the heading change and the historical direction of movement of the terminal when the camera starts capturing the plurality of continuous frames of images.

[0010] In one embodiment, before acquiring multiple frames of images continuously captured by a terminal camera, the receiving module is further used to: determine that a motion state data value of the terminal is less than a threshold value, and the motion state data includes at least acceleration and angular acceleration.

[0011] In one embodiment, the processing module determines the heading change in real time based on the continuous multi-frame images in the following manner: feature points of the same object are marked in the continuous multi-frame images, and there are one or more corresponding feature points in the images of adjacent frames in the continuous multi-frame images, and the corresponding feature points represent the relative position of the same object in the images of adjacent frames in the continuous multi-frame images; according to the relative position of the corresponding feature points in the images of adjacent frames in the continuous multi-frame images, the relative offset of the corresponding feature points is determined in real time; and the heading change is determined based on the relative offset of the corresponding feature points.

[0012] In one embodiment, the processing module determines the heading change based on the relative offset of the corresponding feature point in the following manner: determining a first heading change in real time through an inertial sensor in the terminal; using the relative offset as a second heading change, the relative offset being determined in real time based on the relative position of the corresponding feature point in adjacent frames of the continuous multiple frames; and merging the first heading change with the second heading change according to a preset weight to obtain the heading change.

[0013] In one embodiment, the processing module determines the current movement direction of the terminal based on the heading change and the movement direction of the terminal when starting to acquire the continuous multiple frames of images in the following manner: determining the historical movement direction of the terminal when the camera starts to shoot the continuous multiple frames of images; and superimposing the historical movement direction of the terminal and the heading change in real time through Kalman filtering to obtain the current movement direction of the terminal.

[0014] According to a third aspect of an embodiment of the present disclosure, a navigation and positioning device is provided, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to: execute the method described in the first aspect or any one of the embodiments of the first aspect.

[0015] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor of a terminal, the terminal is enabled to execute the method described in the first aspect or any one of the implementations of the first aspect.

[0016] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: determining the change in the direction of motion through a picture, and then determining the direction of motion, thereby improving the accuracy of determining the direction of motion of pedestrians.

[0017] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0019] Figure 1 It is a principle block diagram of a pedestrian integrated navigation algorithm according to an exemplary embodiment.

[0020] Figure 2 The figure is a flowchart of a navigation and positioning method according to an exemplary embodiment.

[0021] Figure 3 The present invention is a flowchart showing a method of triggering the determination of a motion direction according to an image according to an exemplary embodiment.

[0022] Figure 4 The present invention is a flow chart showing a method of determining a heading change amount according to an image according to an exemplary embodiment.

[0023] Figure 5 The present invention is a flow chart showing a method of determining a heading change amount based on corresponding feature points according to an exemplary embodiment.

[0024] Figure 6 The present invention is a flow chart showing a method of determining a movement direction according to a historical movement direction and a heading change according to an exemplary embodiment.

[0025] Figure 7 The figure is a schematic diagram showing a method of obtaining a heading change based on feature recognition of a normally-on camera according to an exemplary embodiment.

[0026] Figure 8 The figure is a schematic diagram showing a method of adding heading change amount to assist combined positioning according to an exemplary embodiment.

[0027] Fig. 9 The figure is a block diagram of a navigation and positioning device according to an exemplary embodiment.

[0028] Fig.10 The invention is a block diagram of a device for navigation and positioning according to an exemplary embodiment. DETAILED DESCRIPTION

[0029] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0030] Integrated navigation technology is generally an algorithm that uses the data of the Inertial Measurement Unit (IMU), maps, and other data to fuse with the Global Navigation Satellite System (GNSS) data to optimize the positioning results. Among them, the fusion of IMU and GNSS is the most commonly used. Due to its high dynamics, pedestrian integrated navigation generally uses the integration method to solve the IMU data, calculate the pedestrian's heading angle and speed, and participate in the fusion of GNSS data to determine the current direction of movement of the pedestrian and achieve movement direction positioning. Figure 1As shown, it is a block diagram of the principle of a pedestrian combined navigation algorithm according to an exemplary embodiment. Pedestrian information such as position, speed, and posture is obtained through an accelerometer, a gyroscope, and a magnetometer, and the obtained pedestrian information is integrated through the integration method to determine the position, speed, posture, etc. of the pedestrian. Among them, the posture information includes the turning speed of the pedestrian, etc. Through the quasi-stationary point method, that is, using the gait characteristics of the pedestrian during walking to set a threshold to capture the quasi-stationary point, identify the heading, that is, determine the direction of movement of the pedestrian. After obtaining the position, speed, posture and heading information of the pedestrian, it is input into the Kalman filter with the GNSS data for Kalman filtering, and the pedestrian motion state information of the two is combined to determine the optimized positioning data, thereby realizing pedestrian positioning and navigation.

[0031] However, due to the accuracy of physical devices such as accelerometers, gyroscopes, and magnetometers, when using the gait characteristics of pedestrians to set thresholds for quasi-static point capture, this method is very prone to errors due to the strong dynamics of pedestrians when walking, resulting in negative positioning optimization.

[0032] In related technologies, the surrounding environment can also be photographed by the electronic device camera, and the photographed image can be compared with the image pre-stored in the database to achieve auxiliary positioning. However, this method requires the collection of environmental data in advance, which is costly, and the image is not taken in real time, which is easy to interfere with the image comparison result and affect the final positioning effect.

[0033] In view of this, the present disclosure provides a navigation positioning method. A camera continuously captures images, and the change amount of the current motion direction is determined in real time according to the captured images. The current motion direction is determined in real time according to the change amount of the current motion direction in real time and the motion direction before the image is captured, thereby improving the accuracy of combined navigation positioning.

[0034] Compared with the positioning method in the related art, the navigation positioning method provided by the present disclosure determines the change in the direction of motion in real time based on the image, and the determination of the change in the direction of motion is more accurate. The following will describe how to determine the direction of motion based on the real-time image.

[0035] Figure 2 is a flowchart of a navigation and positioning method according to an exemplary embodiment. Figure 2 As shown, the navigation positioning method includes the following steps.

[0036] In step S11, a plurality of continuous frames of video captured by a terminal camera are obtained.

[0037] Each frame of the video captured can reflect the changes in the scene in the image over time, from which the required positioning information can be determined to improve the accuracy of positioning.

[0038] In step S12, the heading change amount is determined in real time based on the continuous multiple frames of images, and the heading change amount represents the change amount of the terminal's moving direction.

[0039] In the disclosed embodiment, by determining the change in the terminal's motion direction in real time based on the change in a plurality of consecutive frames of images, the motion direction can be determined more accurately.

[0040] In step S13, the current moving direction of the terminal is determined based on the heading change amount and the historical moving direction of the terminal when the camera starts to capture multiple consecutive frames of images.

[0041] In the disclosed embodiment, the historical movement direction is the terminal movement direction when the image is captured, and the historical direction is recorded as a basis for subsequent terminal movement direction determination.

[0042] In one implementation, multiple images of consecutive frames may be captured instead of capturing multiple consecutive frames of video, wherein the interval between consecutive frames may be determined by the terminal.

[0043] In one embodiment, a video is captured, and the heading change of the terminal is determined in real time based on a plurality of consecutive frames of images in the video. The direction of movement of the terminal is determined based on the direction of movement of the terminal when the image is captured and the heading change obtained based on the plurality of consecutive frames of images in the video. By capturing images and determining the heading change in real time, the accuracy is higher than that of only using physical devices, so that the direction of movement of the terminal can be determined more accurately.

[0044] Figure 3 The present invention is a flowchart showing a method of triggering the determination of a motion direction according to an image according to an exemplary embodiment.

[0045] In step S21, it is determined that the motion state data value of the terminal is less than a threshold value, and the motion state data at least includes acceleration and angular acceleration.

[0046] In the disclosed embodiment, the threshold is a value preset by the terminal system. By setting the threshold for triggering shooting, it is prevented that the reference of the captured image is reduced when the terminal moves significantly.

[0047] In step S22, multiple frames of continuous video images captured by the terminal camera are obtained.

[0048] In the embodiment of the present disclosure, step S22 is the same as step S11 in the above embodiment and will not be described again.

[0049] In one implementation, the terminal detects its own motion state data. When the motion state data value is less than a preset threshold, the terminal camera starts to shoot video to obtain a video composed of multiple frames of images for subsequent determination of the terminal's motion direction. By setting a threshold, the terminal starts shooting video in a low-dynamic state to prevent interference from a high-dynamic state, which may cause the captured multiple frames of images to have no reference. Among them, the low-dynamic state is a state in which the terminal motion state data value is lower than a preset threshold. At this time, the physical device of the terminal may not be able to accurately determine the change in the terminal's motion direction due to accuracy issues. The high-dynamic state is a state in which the terminal motion state data value is higher than a preset threshold. At this time, there may be no correlation between the pictures taken by the terminal. Therefore, it is necessary to set a threshold to trigger the shooting of the video in a low-dynamic state, and obtain a video composed of multiple continuous frames of images with strong correlation, so as to more accurately determine the terminal's motion direction.

[0050] Figure 4 The present invention is a flow chart showing a method of determining a heading change amount according to an image according to an exemplary embodiment.

[0051] In step S31, feature points of the same object are marked in the continuous multi-frame images, and there are one or more corresponding feature points in the images of adjacent frames in the continuous multi-frame images, and the corresponding feature points represent the relative position of the same object in the images of adjacent frames in the continuous multi-frame images.

[0052] In the disclosed embodiment, the same object in the continuous multi-frame images may exist in the multi-frame images. The position of the object in the multi-frame images is used as a reference to mark feature points, which can be used to compare the position changes of the same object in different images. Among them, the number of corresponding feature points is random, and the position of the corresponding feature points is determined according to the position of the same object in the image of the adjacent frame in the continuous multi-frame images. The corresponding feature points are feature points corresponding to the same object.

[0053] In step S32, the relative offset of the corresponding feature point is determined in real time according to the relative position of the corresponding feature point in the image of adjacent frames in the continuous multiple frames of images.

[0054] In the disclosed embodiment, the position change of corresponding feature points in adjacent frame images represents the relative position of the same object in the adjacent images, and also represents the change of the terminal's motion direction.

[0055] In step S33, the heading change is determined based on the relative offset of the corresponding feature point.

[0056] In the disclosed embodiment, the relative offset of the corresponding feature point represents the change of the terminal shooting direction, and further represents the change of the terminal movement direction. And the heading change, i.e. the change of the terminal movement direction, is determined by the relative offset of the corresponding feature point in the image, with higher accuracy.

[0057] In one embodiment, after acquiring multiple consecutive frames of images taken by a terminal camera, feature points of the same object in the multiple consecutive frames are marked. According to the relative positions of feature points corresponding to the same object in adjacent frames of images, the position offset of the corresponding feature points in adjacent frames is determined. According to the position offset of the corresponding feature points, the change in the terminal motion direction is determined. The change in the terminal motion direction is determined by the relative offset of the corresponding feature points in the image, which is more accurate than the traditional method.

[0058] In one embodiment, when marking feature points of an image, the terminal system will automatically determine the main image of the object in the image and mark the feature points based on the main image. When the electronic device moves, the main image position of the object in the image will also change. The change in the main image position can be determined based on the relative position of the marked feature points, and then the change in the direction of movement can be determined. For example, if positioning is required on a playground track, the track line can be marked with feature points, and the direction of the current pedestrian's movement can be determined based on the position change of the track line in multiple consecutive frames of images, so as to achieve accurate positioning.

[0059] In one implementation, feature points can be identified through visual simultaneous localization and mapping (VSLAM), the position offsets of corresponding feature points in adjacent frames of multiple consecutive frames of images captured can be determined, and the heading change can be determined based on the position offsets of the corresponding feature points.

[0060] Figure 5 The present invention is a flow chart showing a method of determining a heading change amount based on corresponding feature points according to an exemplary embodiment.

[0061] In step S41, a first heading change amount is determined in real time by an inertial sensor in the terminal.

[0062] In the disclosed embodiment, the first heading change amount is the change amount of the terminal motion direction determined by the inertial sensor, and is used as a reference amount for determining the change of the terminal motion direction.

[0063] In step S42, the relative offset is used as the second heading change amount, and the relative offset is determined in real time according to the relative positions of the corresponding feature points in the images of adjacent frames in the continuous multiple frames of images.

[0064] In the disclosed embodiment, the first heading change amount is the terminal motion direction change amount determined by the relative positions of corresponding feature points in a plurality of consecutive image frames, and is used to determine the terminal motion direction change in combination with the first heading change amount.

[0065] In step S43, the first heading change amount and the second heading change amount are combined according to a preset weight to obtain a heading change amount.

[0066] In the disclosed embodiment, the heading change amount, that is, the terminal movement direction change amount, is determined in real time based on the first heading change amount and the second heading change amount.

[0067] In one embodiment, the first heading change amount and the second heading change amount in the above embodiment can be combined according to a preset weight by a Kalman filter to determine the terminal movement direction. And the first heading change amount or the second course change amount can both be 0, that is, non-existent. If the first heading change amount or the second course change amount does not exist, the terminal movement direction change amount is determined only based on the existing first heading change amount or the second course change amount.

[0068] In one embodiment, the first heading change is determined in real time by an inertial sensor in the terminal. The first heading change is combined with the terminal motion direction change determined by the relative position of the corresponding feature points in the continuous multi-frame image according to a preset weight to determine the terminal motion direction change. By combining the terminal motion direction change obtained by the inertial sensor and the image feature points, the terminal motion direction change is finally determined to improve the accuracy of positioning the motion direction.

[0069] Figure 6 The present invention is a flow chart showing a method of determining a movement direction according to a historical movement direction and a heading change according to an exemplary embodiment.

[0070] In step S51, the historical movement direction of the terminal when the camera starts to capture multiple consecutive frames of images is determined.

[0071] In the disclosed embodiment, the historical motion direction at the start of image capture is used as basic data for determining the current motion direction of the terminal.

[0072] In step S52, the historical movement direction and heading change of the terminal are superimposed in real time through Kalman filtering to obtain the current movement direction of the terminal.

[0073] In the disclosed embodiment, based on the historical movement direction data, the current movement direction of the terminal in the current frame can be determined by superimposing the heading change in real time.

[0074] In one implementation, the terminal movement direction at the start of image capture is determined, and the terminal movement direction at the start of image capture is superimposed on the heading change determined in real time, and the superimposed movement direction is used as the current movement direction of the terminal. By updating the terminal movement change determined by the image in real time, and then determining the information of the current movement direction of the terminal in real time, the accuracy of the determination of the current movement direction of the terminal is improved.

[0075] In one implementation, without considering the power consumption of the terminal, the terminal camera can be kept in a normally-on state, and shooting can be performed when in a low-motion mode.

[0076] In one implementation, the terminal camera can be turned off, and when in a low-motion mode, the camera can be turned on and shooting can be performed, thereby saving power consumption of the terminal.

[0077] In one implementation, an always-on camera (AON camera) may be used to capture images. Compared with a traditional camera, an AON camera consumes less power in standby mode, and consumes less power in both the always-on and closed states than other types of cameras in the corresponding states, thereby increasing the battery life of the electronic device.

[0078] Figure 7 The figure is a schematic diagram showing a method of obtaining a heading change based on feature recognition of a normally-on camera according to an exemplary embodiment.

[0079] In one embodiment, if Figure 7 As shown, the terminal motion state data can be collected in real time by the IMU device of the terminal. The value of the angular acceleration in the IMU data is compared with the preset threshold, and pattern recognition is performed to determine the current terminal motion state. Among them, the angular acceleration can also be determined by other methods other than the IMU device, but considering that the IMU device has the best performance in obtaining angular acceleration and other positioning data, the IMU device is used in the embodiment of the present disclosure to measure the terminal angular acceleration. When the value of the angular acceleration in the IMU data is greater than the preset threshold, it is determined that the terminal is not in low dynamic mode at this time, and the angular acceleration data continues to be obtained and pattern recognition is performed. When the value of the angular acceleration in the IMU data is less than or equal to the preset threshold, it indicates that the current terminal is in low dynamic mode. At this time, the AON camera can automatically and continuously capture images of the surrounding environment to generate AON camera data in the form of video. Among them, the AON camera consumes less energy than traditional cameras, and other types of cameras can also be used to capture images to obtain corresponding video data. After the video data is obtained, feature points are marked in each frame of the video, and feature points are recognized. Among them, the marked feature points represent the position of the object body captured in each frame of the image in the image. VSLAM can be used to identify feature points, determine the position offset of corresponding feature points in adjacent frame images in the video, and determine the heading change according to the position offset. The heading change of the terminal is used as the heading change of the pedestrian for positioning the pedestrian.

[0080] Figure 8 The figure is a schematic diagram showing a method of adding heading change amount to assist combined positioning according to an exemplary embodiment.

[0081] In one embodiment, if Figure 8 As shown, the motion state data of the pedestrian is obtained in real time through the accelerometer, gyroscope and magnetometer, and the motion state data is solved by the dead reckoning (DR) algorithm to determine the direction and speed of the pedestrian. The heading change is obtained by the method in the embodiment of the present disclosure, and the obtained heading change, the heading and speed obtained by the DR method and the GNSS data are input into the Kalman filter for fusion, and the optimized positioning data is output to determine the real-time position and motion state of the pedestrian. Among them, the GNSS data includes at least the pedestrian position information and the motion direction information.

[0082] The positioning and navigation method provided by the present disclosure determines the heading offset according to the position of the corresponding feature points in the continuous multi-frame images in the captured video, which is more accurate than the traditional positioning method in monitoring the heading change. The camera is called to shoot images when the preset low-dynamic mode is met, which solves the error in the direction of motion judgment caused by the accuracy of physical devices in the traditional positioning method. The more accurate heading change amount obtained is combined with the conventional positioning method to determine the final positioning data, and the positioning is more accurate.

[0083] Based on the same concept, an embodiment of the present disclosure also provides a navigation and positioning device.

[0084] It is understandable that the navigation and positioning device provided by the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of each example disclosed in the embodiment of the present disclosure, the embodiment of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiment of the present disclosure.

[0085] Fig. 9 is a block diagram of a navigation and positioning device according to an exemplary embodiment. Fig. 9 The device includes a receiving module 101 and a processing module 102.

[0086] The receiving module 101 is configured to obtain multiple frames of images continuously captured by a terminal camera, where the continuous multiple frames of images include continuous multiple frames of images in a captured video and multiple frames of images captured in continuous frames.

[0087] The processing module 102 is configured to determine the heading change in real time based on multiple frames of images, where the heading change represents the change in the terminal's movement direction; and determine the terminal's current movement direction based on the heading change and the terminal's historical movement direction when the camera starts taking multiple frames of images.

[0088] In one implementation of the disclosed embodiment, before acquiring multiple frames of images continuously captured by the terminal camera, the receiving module 101 is further configured to determine that the motion state data value of the terminal is less than a threshold value, and the motion state data includes at least acceleration and angular acceleration.

[0089] In one implementation of the disclosed embodiment, the processing module 102 determines the heading change in real time based on multiple frames of images in the following manner: feature points are marked on the same object in consecutive multiple frames of images, and there are one or more corresponding feature points in each frame of adjacent frames in the consecutive multiple frames of images, and the corresponding feature points represent the relative position of the same object in each frame of adjacent frames in the consecutive multiple frames of images; the relative offset of the corresponding feature point is determined in real time based on the relative position of each frame of image and the corresponding feature point in the adjacent image taken later; and the heading change is determined based on the relative offset of the corresponding feature point.

[0090] In one implementation of the disclosed embodiment, the processing module 102 determines the heading change based on the relative offset of the corresponding feature point in the following manner: determining the first heading change in real time through the inertial sensor in the terminal; determining the relative offset of the corresponding feature point in real time according to the relative position of the corresponding feature point in each frame image and the next frame image, and using the relative offset as the second heading change; merging the first heading change and the second heading change according to a preset weight to obtain the heading change.

[0091] In one implementation of the disclosed embodiment, the processing module 102 determines the current movement direction of the terminal based on the heading change and the historical movement direction of the terminal when the terminal starts to acquire multiple frames of images in the following manner: determining the historical movement direction of the terminal when the camera starts to shoot multiple frames of images; superimposing the historical movement direction of the terminal and the heading change in real time through Kalman filtering to obtain the current movement direction of the terminal.

[0092] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0093] Fig.10 1 is a block diagram of a device 200 for navigation and positioning according to an exemplary embodiment. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0094] Reference Fig.10 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0095] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0096] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0097] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0098] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0099] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0100] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0101] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 may also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 may also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0102] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0103] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0104] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0105] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0106] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0107] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0108] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0109] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. This application is intended to cover any modifications, uses or adaptations of the present disclosure, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0110] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A navigation and positioning method, characterized in that: include: Get multiple frames of video captured by the terminal camera; Determine in real time a heading change amount according to the continuous multiple frames of images, wherein the heading change amount represents a change amount of the terminal's motion direction; The current movement direction of the terminal is determined based on the heading change and the historical movement direction of the terminal when the camera starts to capture the continuous multiple frames of images.

2. The method according to claim 1, characterized in that Before obtaining the continuous multiple-frame images of the video captured by the terminal camera, the method further includes: It is determined that a motion state data value of the terminal is less than a threshold value, wherein the motion state data at least includes acceleration and angular acceleration.

3. The method according to claim 1, characterized in that Determining the heading change in real time according to the continuous multiple frames of images includes: Marking feature points of the same object in the continuous multiple frames of images, where there are one or more corresponding feature points in the images of adjacent frames in the continuous multiple frames of images, and the corresponding feature points represent the relative positions of the same object in the images of adjacent frames in the continuous multiple frames of images; Determining in real time the relative offset of the corresponding feature point according to the relative position of the corresponding feature point in the image of adjacent frames in the continuous multiple frames of images; The heading change is determined based on the relative offset of the corresponding feature point.

4. The method according to claim 3, characterized in that: The determining the heading change amount based on the relative offset of the corresponding feature point includes: Determine a first heading change in real time by using an inertial sensor in the terminal; The relative offset is used as a second heading change amount, and the relative offset is determined in real time according to the relative positions of the corresponding feature points in the images of adjacent frames in the continuous multiple frames of images; The first heading change amount and the second heading change amount are combined according to a preset weight to obtain the heading change amount.

5. The method according to claim 1, characterized in that The determining the current movement direction of the terminal based on the heading change amount and the historical movement direction of the terminal when the terminal starts to acquire the continuous multiple frames of images includes: Determine the historical movement direction of the terminal when the camera starts capturing the continuous multiple frames of images; The historical movement direction of the terminal and the heading change are superimposed in real time through Kalman filtering to obtain the current movement direction of the terminal.

6. A navigation and positioning device, characterized in that: include: A receiving module is used to obtain continuous multi-frame images of the video captured by the terminal camera; A processing module, used to determine a heading change amount in real time according to the continuous multiple frames of images, wherein the heading change amount represents a change amount of the terminal's motion direction; The current movement direction of the terminal is determined based on the heading change and the historical movement direction of the terminal when the camera starts to capture the continuous multiple frames of images.

7. The device according to claim 6, characterized in that Before obtaining the continuous multi-frame images of the video captured by the terminal camera, the receiving module is also used to: It is determined that a motion state data value of the terminal is less than a threshold value, wherein the motion state data at least includes acceleration and angular acceleration.

8. The device according to claim 6, characterized in that The processing module determines the heading change in real time according to the multiple frames of images in the following manner: Marking feature points of the same object in the continuous multiple frames of images, where there are one or more corresponding feature points in the images of adjacent frames in the continuous multiple frames of images, and the corresponding feature points represent the relative positions of the same object in the images of adjacent frames in the continuous multiple frames of images; Determining in real time the relative offset of the corresponding feature point according to the relative position of the corresponding feature point in the image of adjacent frames in the continuous multiple frames of images; The heading change is determined based on the relative offset of the corresponding feature point.

9. The device according to claim 8, characterized in that The processing module determines the heading change amount based on the relative offset of the corresponding feature point in the following manner: Determine a first heading change in real time by using an inertial sensor in the terminal; The relative offset is used as a second heading change amount, and the relative offset is determined in real time according to the relative positions of the corresponding feature points in the images of adjacent frames in the continuous multiple frames of images; The first heading change amount and the second heading change amount are combined according to a preset weight to obtain the heading change amount.

10. The device according to claim 6, characterized in that The processing module determines the current moving direction of the terminal based on the heading change amount and the historical moving direction of the terminal when the terminal starts to acquire the continuous multiple frames of images in the following manner: Determine the historical movement direction of the terminal when the camera starts capturing the continuous multiple frames of images; The historical movement direction of the terminal and the heading change are superimposed in real time through Kalman filtering to obtain the current movement direction of the terminal.

11. A navigation and positioning device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to: execute the method described in any one of claims 1 to 5.

12. A storage medium, characterized in that: The storage medium stores instructions, and when the instructions in the storage medium are executed by a processor of the terminal, the terminal is enabled to execute the method according to any one of claims 1 to 5.

Images