Indoor navigation method, device and electronic equipment

Abstract

The embodiment of the present invention provides an indoor navigation method, device and electronic device, wherein the method comprises: obtaining the posture change and speed change of the follower device at the current moment relative to the previous moment based on the relative position relationship of the inertial measurement unit IMU pre-integration and matching feature points; obtaining a matching navigation path based on the posture change and the speed change, and guiding the follower to the target location based on the navigation path; the inertial measurement unit IMU pre-integration is the integration result of the IMU measurement value detected by the follower device in the time period from the previous moment to the current moment, and the matching feature point is the feature point successfully matched in the image collected at the current moment and the previous moment respectively. The embodiment of the present invention can effectively solve the problem of monocular vision scale uncertainty by integrating IMU and visual measurement and providing a unified scale for monocular vision with the help of IMU.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to an indoor navigation method, device and electronic equipment. Background Art

[0002] As the scale of indoor environments continues to grow and their layouts become increasingly complex, indoor navigation is becoming an increasingly basic service in scenarios such as large supermarkets and warehouse management.

[0003] In recent years, with the improvement of computing power of mobile devices and the maturity of Visual Simultaneous Localization and Mapping (Visual SLAM) technology, visual navigation has become more and more popular due to its higher accuracy and richer semantic information. Currently, vision-based point-to-point navigation systems have emerged.

[0004] However, since mainstream devices can only support monocular image applications, current purely vision-based point-to-point navigation faces the challenge of monocular vision scale uncertainty when merging maps. Summary of the invention

[0005] The embodiments of the present invention provide an indoor navigation method, device and electronic device, which are used to solve the defect of scale uncertainty in indoor navigation using monocular vision technology in the prior art, and achieve the goal of unifying the calculation scale during the navigation process.

[0006] An embodiment of the present invention provides an indoor navigation method, including:

[0007] Based on the pre-integration of the inertial measurement unit (IMU) and the relative position relationship of the matching feature points, the position and speed changes of the follower device at the current moment relative to the previous moment are obtained;

[0008] Based on the position change and the speed change, a matching navigation path is acquired, and based on the navigation path, the follower is guided to a target location;

[0009] The inertial measurement unit IMU pre-integration is the integration result of the IMU measurement value detected by the follower device in the time period from the previous moment to the current moment, and the matching feature point is the feature point successfully matched in the images collected at the current moment and the previous moment respectively.

[0010] According to an indoor navigation method of an embodiment of the present invention, before acquiring the position change and speed change of the follower device at the current moment relative to the previous moment, the method further includes:

[0011] Using a monocular camera to collect a first image of the surrounding environment of the follower device, and based on the IMU, obtaining an IMU measurement value of the movement process of the follower device;

[0012] The IMU measurement values ​​are aligned according to the same timestamp, and the IMU measurement values ​​after the alignment are integrated within the time period from the previous moment to the current moment to obtain the IMU pre-integral of the inertial measurement unit;

[0013] Feature points are extracted from the first image collected at the current moment and the previous moment respectively, and matching operations are performed on the extracted feature points to obtain successfully matched feature points in the first image corresponding to the current moment and the previous moment respectively.

[0014] The indoor navigation method according to one embodiment of the present invention further includes:

[0015] Based on the first image collected from the previous moment to the current moment, the optical flow method is used for tracking to obtain the feature point matching results of two adjacent frames of images, and based on the feature point matching results, a random sampling consistency check method based on the basic matrix is ​​used to verify the successfully matched feature points.

[0016] According to an indoor navigation method of an embodiment of the present invention, acquiring a matching navigation path based on the position change and the speed change includes:

[0017] Based on the position change, the speed change and the first image of the surrounding environment collected at the current moment, a matching operation is performed with the pre-stored motion trajectory data of the navigator to obtain the navigation path.

[0018] The indoor navigation method according to one embodiment of the present invention further includes:

[0019] Based on the motion trajectory of the follower device and the pre-stored motion trajectories of all navigators, repositioning detection is performed and trajectory splicing and optimization are performed to obtain a global map of the navigation area.

[0020] The indoor navigation method according to one embodiment of the present invention further includes:

[0021] Collect second pictures of landmark buildings at the starting point and the end point of the navigator's movement track, and add the second pictures to the global map.

[0022] According to an indoor navigation method of an embodiment of the present invention,

[0023] Using the follower device, the IMU measurement value and the image of the environment around the follower device are collected, and feature point matching operation is performed on the images collected at the current moment and the previous moment, and the matching feature points are obtained and uploaded to the cloud;

[0024] Utilizing the cloud, calculating the inertial measurement unit IMU pre-integration based on the IMU measurement value, and calculating the relative position relationship based on the matching feature points, and, based on the inertial measurement unit IMU pre-integration and the relative position relationship, executing the steps of obtaining the posture change and speed change of the follower device at the current moment relative to the previous moment to obtaining the matching navigation path, and sending the navigation path to the follower device;

[0025] The follower device is used to guide the follower to a target location based on the navigation path.

[0026] An embodiment of the present invention further provides an indoor navigation device, comprising:

[0027] The calculation module is used to obtain the posture change and speed change of the follower device at the current moment relative to the previous moment based on the pre-integration of the inertial measurement unit IMU and the relative position relationship of the matching feature points;

[0028] A navigation module, configured to obtain a matching navigation path based on the position change and the speed change, and guide the follower to a target location based on the navigation path;

[0029] The inertial measurement unit IMU pre-integration is the integration result of the IMU measurement value detected by the follower device in the time period from the previous moment to the current moment, and the matching feature point is the feature point successfully matched in the images collected at the current moment and the previous moment respectively.

[0030] An embodiment of the present invention further provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the steps of any of the indoor navigation methods described above are implemented.

[0031] An embodiment of the present invention further provides a non-transitory computer-readable storage medium on which computer instructions are stored. When the computer instructions are executed by a computer, the steps of any of the indoor navigation methods described above are implemented.

[0032] The indoor navigation method, device and electronic device provided by the embodiments of the present invention can effectively solve the problem of monocular vision scale uncertainty and achieve the unification of calculation scales during navigation by integrating IMU and visual measurement and using IMU to provide a unified scale for monocular vision. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 A schematic diagram of a flow chart of an indoor navigation method provided by an embodiment of the present invention;

[0035] Figure 2 A schematic diagram of a system architecture for an indoor navigation method application provided by an embodiment of the present invention;

[0036] Figure 3 A schematic diagram of a flow chart of an indoor navigation method provided by another embodiment of the present invention;

[0037] Figure 4 A schematic diagram showing a comparison of the positioning error CDF of the indoor navigation method provided by an embodiment of the present invention and some existing navigation methods;

[0038] Figure 5 A schematic diagram of the structure of an indoor navigation device provided by an embodiment of the present invention;

[0039] Figure 6 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] The embodiments of the present invention aim at the problem of scale uncertainty in indoor navigation using monocular vision technology in the prior art. By integrating IMU and visual measurement, the IMU is used to provide a unified scale for monocular vision, which can effectively solve the problem of scale uncertainty in monocular vision and achieve the unification of calculation scales during navigation. The embodiments of the present invention will be specifically described and introduced through multiple embodiments below.

[0042] Figure 1 A flowchart of an indoor navigation method provided by an embodiment of the present invention is shown in FIG. Figure 1 As shown, the method includes:

[0043] S101, based on the inertial measurement unit IMU pre-integration and the relative position relationship of the matching feature points, obtain the posture change and speed change of the follower device at the current moment relative to the previous moment.

[0044] Among them, the inertial measurement unit IMU pre-integration is the integration result of the IMU measurement value detected by the follower device in the time period from the previous moment to the current moment, and the matching feature point is the feature point successfully matched in the images collected at the current moment and the previous moment respectively.

[0045] It can be understood that the embodiment of the present invention uses visual inertial odometry (VIO) to record the trajectory of a single user and perform positioning. Visual inertial odometry uses a monocular camera as an external source, and the sensor continuously takes pictures; it uses an inertial measurement unit (IMU) as an internal source, and the sensor collects high-frequency acceleration and angular velocity values.

[0046] The changes in the device's posture and velocity at two adjacent moments can be constrained by the numerical integration results of the IMU measurements during this short period of time (called pre-integration) and the relative position relationship of the successfully matched feature points in the images collected at these two moments. Based on the above constraints, the device's posture and velocity changes that meet these constraints can be obtained using nonlinear optimization methods.

[0047] S102, acquiring a matching navigation path based on the position change and the speed change, and guiding the follower to a target location based on the navigation path.

[0048] It should be understood that the trajectory of each user (including the navigator and the follower) includes a series of device postures during the user's movement, screened images with a certain parallax between each other (called key frames), and a series of three-dimensional points in space corresponding to the successfully tracked feature points in the images (i.e., the map in SLAM terminology).

[0049] Based on the position and speed changes of the follower device calculated according to the above steps, some key frames can be obtained by feature matching based on the pictures of the surrounding environment taken by the follower device during movement, and these key frames are matched with the key frames in the pre-stored trajectory of the navigator. Based on the matching results, matching operations of the device position and speed changes are performed to finally obtain the navigation trajectory of the follower from the starting point to the destination.

[0050] Afterwards, the motion trajectory of the matched leader can be fed back to the follower device, and the follower can be guided to the destination through the follower device, such as by displaying a navigation map or voice navigation to guide the user to the destination.

[0051] The indoor navigation method provided by the embodiment of the present invention can effectively solve the problem of monocular vision scale uncertainty by integrating IMU and visual measurement, and realize the unification of calculation scale in the navigation process by using IMU to provide a unified scale for monocular vision.

[0052] It should be understood that merging the trajectories of different leaders depends on the loop detection technology in SLAM, which is a relatively mature method for detecting whether several pictures that have no temporal dependency are taken of the same place. After the loop detection, a process called pose graph optimization is usually performed to optimize the entire trajectory.

[0053] However, loop closure detection requires the construction of a relatively large keyframe database, and mobile devices cannot afford such storage overhead; pose graph optimization involves Jacobi operations on very high-dimensional variables, and the computing power of mobile devices cannot complete it in real time.

[0054] Therefore, according to the indoor navigation method provided in the above embodiments, the follower device is optionally used to collect the IMU measurement values ​​and the image of the surrounding environment of the follower device, and the image collected at the current moment and the previous moment is respectively subjected to feature point matching operation to obtain the matching feature points and upload them to the cloud; the cloud is used to calculate the inertial measurement unit IMU pre-integration based on the IMU measurement values, and the relative position relationship is calculated based on the matching feature points, and, based on the inertial measurement unit IMU pre-integration and the relative position relationship, the steps of obtaining the posture change and speed change of the follower device at the current moment relative to the previous moment to obtain the matching navigation path are performed, and the navigation path is sent to the follower device; the follower device is used to guide the follower to the target location based on the navigation path.

[0055] It can be understood that the embodiment of the present invention not only needs to perform VIO for a single user by integrating vision and IMU measurement, but also needs to stitch the trajectories of multiple users. The mobile device itself cannot provide sufficient computing and storage support. Therefore, the embodiment of the present invention decomposes the entire system into two parts: the front-end and the back-end. Figure 2 As shown in FIG. 1 , a schematic diagram of the system architecture of the indoor navigation method provided by an embodiment of the present invention is shown. The front end is responsible for data collection and preprocessing of a single user, and the back end is responsible for completing VIO optimization and trajectory stitching tasks. On the one hand, it reduces the computing burden of the mobile terminal, and on the other hand, it also significantly saves bandwidth.

[0056] Feature points are extracted on the mobile terminal and tracked using the optical flow method. For the matching results of two adjacent frames obtained by the optical flow method, a random sampling consistency check based on the basic matrix is ​​used to reduce false matches.

[0057] The mobile terminal can transmit the processed feature points at a rate of 30fps. When each image contains 100 feature points, the bandwidth usage is only 180KB / s, which is much lower than the 6.1MB / S of directly transmitting the image at the same rate. Considering that a considerable number of mobile devices can take pictures at a higher rate (such as 60Hz or even 120Hz), the system can only transmit the feature points extracted from a part of the pictures to ensure the minimum picture frequency and perform optical flow tracking on each picture, so as to ensure the quality of the feature points in the transmitted pictures.

[0058] Considering that the IMU pre-integration needs to be repeatedly adjusted during the optimization process as the bias changes, the front end only completes a simple data alignment operation, that is, generating accelerometer and gyroscope readings with the same timestamp.

[0059] With the help of the IMU data and feature points provided by the front-end, the back-end can perform short-term VIO on a single user. A complete trajectory record also requires a series of pictures as key frames, calculating the descriptors of the feature points in these pictures and building a bag-of-words model.

[0060] Considering that key frames are mainly used for repositioning and are not time-sensitive content, the system transmits images as key frames at a lower frequency (1Hz), which also does not put pressure on the bandwidth.

[0061] In terms of system implementation, the embodiment of the present invention divides the entire system into a front end responsible for image and IMU data preprocessing and a back end responsible for trajectory fusion and optimization. The front end runs on a mobile device and the back end runs on the cloud, which effectively reduces the computing burden of the mobile device and ensures real-time performance while establishing a globally consistent map.

[0062] Among them, according to the indoor navigation method provided by the above embodiments, optionally, before obtaining the posture change and speed change of the follower device at the current moment relative to the previous moment, it also includes: using a monocular camera to collect a first image of the surrounding environment of the follower device, and based on the IMU, obtaining the IMU measurement value of the movement process of the follower device; aligning the IMU measurement values ​​according to the same timestamp, and integrating the aligned IMU measurement values ​​in the time period from the previous moment to the current moment to obtain the IMU pre-integral; extracting feature points of the first images collected at the current moment and the previous moment respectively, and matching the extracted feature points to obtain the successfully matched feature points in the first images corresponding to the current moment and the previous moment respectively.

[0063] Specifically, if Figure 3 As shown, it is a flow chart of an indoor navigation method provided by another embodiment of the present invention, wherein the embodiment of the present invention needs to calculate the relative position relationship between the IMU pre-integration and the matching feature points before performing navigation operations according to the above embodiments. The embodiment of the present invention uses the visual-inertial odometer VINS-Mono at the front end to record the trajectory of a single user and perform positioning. The visual-inertial odometer uses a monocular camera as an external source, and the sensor continuously takes pictures, and the image obtained can be called the first image; using IMU as an internal source, the sensor collects high-frequency acceleration and angular velocity values, which can be called IMU measurement values.

[0064] After that, feature point matching is performed based on the collected images to obtain matching feature points. At the same time, an integral operation is performed based on the IMU measurement value in a short period of time from the previous moment to the current moment to obtain the IMU pre-integral.

[0065] Among them, the indoor navigation method provided according to the above embodiments may optionally also include: based on the first image collected from the previous moment to the current moment, tracking is performed using the optical flow method to obtain feature point matching results of two adjacent frames of images, and based on the feature point matching results, a random sampling consistency check method based on the basic matrix is ​​used to verify the successfully matched feature points.

[0066] It can be understood that after matching the feature points of the first image collected, the validity of the matched feature points can be further verified. That is, feature points are extracted on the mobile terminal and tracked using the optical flow method. For the matching results of two adjacent frames obtained by the optical flow method, a random sampling consistency check based on the basic matrix is ​​used to reduce mismatching.

[0067] Among them, according to the indoor navigation method provided in the above embodiments, optionally, obtaining a matching navigation path based on the posture change and the speed change includes: performing a matching operation with pre-stored navigator's motion trajectory data based on the posture change, the speed change and the first image of the surrounding environment collected at the current moment to obtain the navigation path.

[0068] It can be understood that in the embodiment of the present invention, when a new user opens the camera and starts running the system, the system will search in all key frames of the stored trajectory based on the currently captured picture, and infer the user's position on the global map with the help of descriptor matching and bag-of-words model. During the user's movement, the system will continuously compare the pictures taken by the user with the pictures that have been saved to find new feature point correspondences, and all correspondences are used to solve the optimization problem:

[0069]

[0070] Among them, R, t represents the rotation and translation of the local map built by the user with the help of VIO to the global map, and the sequence pair (l, v) represents the lth feature observed in the vth key frame found. represents its observed value, Indicates the location of the map point corresponding to the observation result in the global map.

[0071] The goal of the optimization problem is to find a coordinate transformation that minimizes the difference between the global map and the map observed by the user. With this coordinate transformation, the user's trajectory can be merged into the global map.

[0072] Among them, the indoor navigation method provided according to the above embodiments may optionally further include: performing repositioning detection and trajectory splicing and optimization based on the motion trajectory of the follower device and the motion trajectory of all pre-stored navigators to obtain a global map of the navigation area.

[0073] It can be understood that in the embodiment of the present invention, the follower turns on the camera at the starting point and moves appropriately, and the system locates the follower in the global map by relocation. That is, first find the track where the follower's target is located, and according to the track splicing relationship in the current global map, determine which tracks provided by other leaders must be passed between the leader's track containing the current user's position and the target track.

[0074] The system further corrects the spliced ​​navigation path based on the distance between key frames in different trajectories, removing possible detours to obtain a path that is as short as possible.

[0075] Once the system detects that the current scene corresponds to the previous scene, it will eliminate the drift of the accumulated position through relocalization. After the follower reaches the destination, its trajectory will also be added to the global map to further enrich the keyframe database.

[0076] In other words, the cloud will relocate each newly added track with all previously stored tracks, so that a newly added track can add multiple tracks that have not been merged before to the global map.

[0077] like Figure 4 As shown, it is a schematic diagram of the comparison of the positioning error CDF of the indoor navigation method provided by the embodiment of the present invention and some existing navigation methods. In the embodiment of the present invention, VIO is always running during the movement of the follower. Since the user has been positioned in the global map when starting to use the system, even in the absence of repositioning (for example, when the follower moves in the opposite direction of the trajectory provided by the leader), the user can have a relatively accurate estimate of the current position, so the navigation calculation is more accurate and the effect is better.

[0078] It should be understood that although visual measurement has higher accuracy, it is more sensitive to direction. Even in the same location, if the user faces different directions, there may be completely different visual measurement results. This means that in pure visual navigation, the follower can only follow the leader from the starting point to the end point, but not from the end point to the starting point.

[0079] Therefore, the indoor navigation method provided according to the above embodiments may optionally further include: collecting second pictures of landmark buildings at the starting point and the end point of the navigator's motion trajectory, and adding the second pictures to the global map.

[0080] Specifically, in addition to merging trajectories to create a global map with the help of repositioning automatically detected during movement, an embodiment of the present invention requires the leader to take photos of landmark buildings or facades when departing and arriving at the destination, so that when two trajectories have the same starting point or end point but different directions, they can also be merged into the global map.

[0081] The embodiment of the present invention can fully utilize the overlap between trajectories to construct a global map and navigate followers, and has stronger applicability.

[0082] Based on the same inventive concept, an embodiment of the present invention provides an indoor navigation device according to the above embodiments, and the device is used to implement indoor navigation in the above embodiments. Therefore, the description and definition in the indoor navigation method in the above embodiments can be used for understanding each execution module in the embodiment of the present invention, and the above embodiments can be referred to for details, which will not be repeated here.

[0083] According to one embodiment of the present invention, the structure of the indoor navigation device is as follows: Figure 5 As shown, it is a schematic diagram of the structure of an indoor navigation device provided by an embodiment of the present invention, and the device can be used to implement indoor navigation in the above-mentioned method embodiments, and the device includes: a calculation module 501 and a navigation module 502. Among them:

[0084] The calculation module 501 is used to obtain the posture change and speed change of the follower device at the current moment relative to the previous moment based on the relative position relationship of the inertial measurement unit IMU pre-integration and matching feature points; the navigation module 502 is used to obtain the matching navigation path based on the posture change and the speed change, and guide the follower to the target location based on the navigation path.

[0085] Among them, the inertial measurement unit IMU pre-integration is the integration result of the IMU measurement value detected by the follower device in the time period from the previous moment to the current moment, and the matching feature point is the feature point successfully matched in the images collected at the current moment and the previous moment respectively.

[0086] Specifically, the position and velocity changes of the device at two adjacent moments in the embodiment of the present invention can be constrained by the numerical integration results of the IMU measurement values ​​in this short period of time (called inertial measurement unit IMU pre-integration) and the relative position relationship of the successfully matched feature points in the images collected at these two moments. Based on the above constraint relationship, the calculation module 501 can use a nonlinear optimization method to calculate the device position and velocity changes that meet these constraints.

[0087] Afterwards, the navigation module 502 can obtain some key frames by feature matching based on the pictures of the surrounding environment taken by the follower device during the movement, and match these key frames with the key frames in the pre-stored leader's trajectory, and based on the matching results, perform matching operations on the device posture changes and speed changes, and finally obtain the navigation trajectory of the follower from the starting point to the destination.

[0088] Afterwards, the navigation module 502 may feed back the matched movement trajectory of the leader to the follower device, and guide the follower to the destination through the follower device, such as by displaying a navigation map or voice navigation to guide the user to the destination.

[0089] The indoor navigation device provided by the embodiment of the present invention can effectively solve the problem of monocular vision scale uncertainty by integrating IMU and visual measurement, and realize the unification of calculation scale during navigation.

[0090] It is understandable that in the embodiment of the present invention, the relevant program modules in the apparatus of each of the above embodiments can be implemented by a hardware processor. Moreover, the indoor navigation apparatus of the embodiment of the present invention utilizes the above program modules to implement the indoor navigation process of each of the above method embodiments. When used to implement indoor navigation in each of the above method embodiments, the beneficial effects produced by the apparatus of the embodiment of the present invention are the same as those of the corresponding method embodiments, and the above method embodiments can be referred to, and will not be repeated here.

[0091] As another aspect of an embodiment of the present invention, this embodiment provides an electronic device according to the above embodiments, and the electronic device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the indoor navigation method described in the above embodiments are implemented.

[0092] Furthermore, the electronic device of the embodiment of the present invention may also include a communication interface and a bus. Figure 6 , is a schematic diagram of the physical structure of an electronic device provided in an embodiment of the present invention, including: at least one memory 601, at least one processor 602, a communication interface 603 and a bus 604.

[0093] Among them, the memory 601, the processor 602 and the communication interface 603 complete mutual communication through the bus 604, and the communication interface 603 is used for information transmission between the electronic device and the follower device; the memory 601 stores a computer program that can be run on the processor 602. When the processor 602 executes the computer program, the steps of the indoor navigation method described in the above embodiments are implemented.

[0094] It can be understood that the electronic device at least includes a memory 601, a processor 602, a communication interface 603 and a bus 604, and the memory 601, the processor 602 and the communication interface 603 form a communication connection with each other through the bus 604, and can complete mutual communication, such as the processor 602 reading the program instructions of the indoor navigation method from the memory 601. In addition, the communication interface 603 can also realize the communication connection between the electronic device and the follower device, and can complete the information transmission between them, such as realizing the reading of image data and IMU data through the communication interface 603.

[0095] When the electronic device is running, the processor 602 calls the program instructions in the memory 601 to execute the methods provided by the above-mentioned method embodiments, for example, including: based on the relative position relationship between the inertial measurement unit IMU pre-integration and matching feature points, obtaining the posture change and speed change of the follower device at the current moment relative to the previous moment; based on the posture change and the speed change, obtaining a matching navigation path, and based on the navigation path, guiding the follower to the target location, etc.

[0096] The program instructions in the above-mentioned memory 601 can be implemented in the form of software functional units and can be stored in a computer-readable storage medium when sold or used as an independent product. Alternatively, all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the above-mentioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (RandomAccess Memory, RAM), disk or optical disk and other media that can store program codes.

[0097] An embodiment of the present invention further provides a non-transitory computer-readable storage medium according to the above embodiments, on which computer instructions are stored. When the computer instructions are executed by a computer, the steps of the indoor navigation method described in the above embodiments are implemented, for example, including: based on the relative position relationship between the pre-integration of the inertial measurement unit IMU and the matching feature points, obtaining the posture change and speed change of the follower device at the current moment relative to the previous moment; based on the posture change and the speed change, obtaining a matching navigation path, and based on the navigation path, guiding the follower to the target location, etc.

[0098] As another aspect of an embodiment of the present invention, this embodiment further provides a computer program product according to the above embodiments, wherein the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, wherein the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the indoor navigation method provided by the above method embodiments, wherein the method includes: obtaining a posture change and a speed change of a follower device at a current moment relative to a previous moment based on the relative position relationship between the inertial measurement unit IMU pre-integration and matching feature points; obtaining a matching navigation path based on the posture change and the speed change, and guiding the follower to the target location based on the navigation path.

[0099] The electronic device, non-transitory computer-readable storage medium, and computer program product provided by the embodiments of the present invention integrate IMU and visual measurement by executing the steps of the indoor navigation method described in the above embodiments, and use the IMU to provide a unified scale for monocular vision, which can effectively solve the problem of monocular vision scale uncertainty and achieve the unification of calculation scales during navigation.

[0100] It is understandable that the embodiments of the apparatus, electronic device, and storage medium described above are merely illustrative, and the units described as separate components may or may not be physically separated, and may be located in one place, or may be distributed on different network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0101] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk, etc., including a number of instructions to enable a computer device (such as a personal computer, a server, or a network device, etc.) to execute the methods described in the above method embodiments or some parts of the method embodiments.

[0102] In addition, it should be understood by those skilled in the art that, in the application documents of the embodiments of the present invention, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0103] In the description of the embodiment of the present invention, a large number of specific details are described. However, it should be understood that the embodiment of the present invention can be practiced without these specific details. In some examples, known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this specification. Similarly, it should be understood that in order to simplify the disclosure of the embodiment of the present invention and help understand one or more of the various inventive aspects, in the above description of the exemplary embodiments of the embodiment of the present invention, the various features of the embodiment of the present invention are sometimes grouped together into a single embodiment, figure or description thereof.

[0104] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the embodiments of the present invention, rather than to limit them. Although the embodiments of the present invention have been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the embodiments of the present invention.

Claims

1. An indoor navigation method, characterized in that: include: Based on the pre-integration of the inertial measurement unit (IMU) and the relative position relationship of the matching feature points, the position and speed changes of the follower device at the current moment relative to the previous moment are obtained; Based on the position change and the speed change, a matching navigation path is acquired, and based on the navigation path, the follower is guided to a target location; The inertial measurement unit IMU pre-integration is the integration result of the IMU measurement value detected by the follower device in the time period from the previous moment to the current moment, and the matching feature point is the feature point successfully matched in the images respectively collected at the current moment and the previous moment; The position change and speed change of the current moment relative to the previous moment are constrained by the IMU pre-integration from the previous moment to the current moment and the relative position relationship of the feature points successfully matched in the images respectively collected at the current moment and the previous moment; and the position change and speed change of the follower device at the current moment relative to the previous moment are obtained based on the IMU pre-integration and the relative position relationship of the matching feature points, including: Based on the constraint relationship, a nonlinear optimization method is used to find the device posture change and speed change that satisfy the constraint relationship; Wherein, acquiring a matching navigation path based on the position change and the speed change, and guiding the follower to a target location based on the navigation path, includes: Key frames are obtained by feature matching based on the pictures of the surrounding environment taken by the follower device during movement, and the key frames are matched with the key frames in the pre-stored trajectory of the navigator. Based on the matching results, matching operations are performed on the device posture changes and speed changes, and finally the navigation trajectory of the follower from the starting point to the destination is obtained.

2. The indoor navigation method according to claim 1, characterized in that: Before obtaining the position change and speed change of the follower device at the current moment relative to the previous moment, the method further includes: Using a monocular camera to collect a first image of the surrounding environment of the follower device, and based on the IMU, obtaining an IMU measurement value of the movement process of the follower device; The IMU measurement values ​​are aligned according to the same timestamp, and the IMU measurement values ​​after the alignment are integrated within the time period from the previous moment to the current moment to obtain the IMU pre-integral of the inertial measurement unit; Feature points are extracted from the first image collected at the current moment and the previous moment respectively, and matching operations are performed on the extracted feature points to obtain successfully matched feature points in the first image corresponding to the current moment and the previous moment respectively.

3. The indoor navigation method according to claim 2, characterized in that: Also includes: Based on the first image collected from the previous moment to the current moment, the optical flow method is used for tracking to obtain the feature point matching results of two adjacent frames of images, and based on the feature point matching results, a random sampling consistency check method based on the basic matrix is ​​used to verify the successfully matched feature points.

4. The indoor navigation method according to claim 1, characterized in that: The obtaining of a matching navigation path based on the posture change and the speed change includes: Based on the position change, the speed change and the first image of the surrounding environment collected at the current moment, a matching operation is performed with the pre-stored motion trajectory data of the navigator to obtain the navigation path.

5. The indoor navigation method according to claim 1, characterized in that: Also includes: Collect second pictures of landmark buildings at the starting point and the end point of the navigator's movement trajectory, and add the second pictures to the global map.

6. The indoor navigation method according to any one of claims 1 to 5, characterized in that: Using the follower device, the IMU measurement value and the image of the environment around the follower device are collected, and feature point matching operation is performed on the images collected at the current moment and the previous moment, and the matching feature points are obtained and uploaded to the cloud; Utilizing the cloud, calculating the inertial measurement unit IMU pre-integration based on the IMU measurement value, and calculating the relative position relationship based on the matching feature points, and, based on the inertial measurement unit IMU pre-integration and the relative position relationship, executing the steps of obtaining the posture change and speed change of the follower device at the current moment relative to the previous moment to obtaining the matching navigation path, and sending the navigation path to the follower device; The follower device is used to guide the follower to a target location based on the navigation path.

7. An indoor navigation device, characterized in that: include: The calculation module is used to obtain the posture change and speed change of the follower device at the current moment relative to the previous moment based on the pre-integration of the inertial measurement unit IMU and the relative position relationship of the matching feature points; A navigation module, configured to obtain a matching navigation path based on the position change and the speed change, and guide the follower to a target location based on the navigation path; The inertial measurement unit IMU pre-integration is the integration result of the IMU measurement value detected by the follower device in the time period from the previous moment to the current moment, and the matching feature point is the feature point successfully matched in the images respectively collected at the current moment and the previous moment; The position change and speed change of the current moment relative to the previous moment are constrained by the IMU pre-integration from the previous moment to the current moment and the relative position relationship of the feature points successfully matched in the images respectively collected at the current moment and the previous moment; and the position change and speed change of the follower device at the current moment relative to the previous moment are obtained based on the IMU pre-integration and the relative position relationship of the matching feature points, including: Based on the constraint relationship, a nonlinear optimization method is used to find the device posture change and speed change that satisfy the constraint relationship; Wherein, acquiring a matching navigation path based on the position change and the speed change, and guiding the follower to a target location based on the navigation path, includes: Key frames are obtained by feature matching based on the pictures of the surrounding environment taken by the follower device during movement, and the key frames are matched with the key frames in the pre-stored trajectory of the navigator. Based on the matching results, matching operations are performed on the device posture changes and speed changes, and finally the navigation trajectory of the follower from the starting point to the destination is obtained.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the indoor navigation method according to any one of claims 1 to 6 are implemented.

9. A non-transitory computer-readable storage medium having computer instructions stored thereon, characterized in that: When the computer instructions are executed by a computer, the steps of the indoor navigation method according to any one of claims 1 to 6 are implemented.

Images