Visual positioning method and related equipment for indoor navigation

Abstract

The embodiments of the present disclosure provide a visual positioning method and apparatus, a computer-readable storage medium, and an electronic device for indoor navigation, and belong to the field of computer and communication technology. The method comprises: obtaining an indoor query image; obtaining feature points of the query image; obtaining a descriptor of the feature points of the query image; querying an indoor map based on the descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image; comparing each feature point of the query image with the feature points in each key image in the set of candidate key images, and filtering the feature points of the query image at optimal and suboptimal ratios to obtain the feature points of the filtered query image; and determining the position and posture when the query image was taken based on the feature points of the filtered query image. The method of the present disclosure can achieve visual positioning indoors.

Description

Technical Field

[0001] The present disclosure relates to the field of computers and communications technology, and in particular to a visual positioning method and apparatus for indoor navigation, a computer-readable storage medium, and an electronic device. Background Art

[0002] With advancements in virtual reality (VR) and augmented reality (AR) technologies, the application of AR technology to smartphone cameras has enabled interactive experiences within shopping malls, including AR guided tours, AR landscapes, AR red envelopes, and AR games. When we enter an unfamiliar environment, especially indoors where global positioning (GPS) signals are inaccurate, it can be difficult to determine our exact location within the building. To achieve a seamless fusion of the real and virtual worlds, it's essential to accurately control the posture and orientation of our mobile devices (smartphones).

[0003] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute prior art known to ordinary technicians in the field. Summary of the Invention

[0004] The embodiments of the present disclosure provide a visual positioning method and apparatus for indoor navigation, a computer-readable storage medium, and an electronic device, which can achieve indoor visual positioning.

[0005] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by practice of the present disclosure.

[0006] According to one aspect of the present disclosure, a visual positioning method for indoor navigation is provided, comprising:

[0007] Get an indoor query image;

[0008] Acquire feature points of the query image;

[0009] Obtaining descriptors of feature points of the query image;

[0010] querying an indoor map according to the descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image;

[0011] Comparing each feature point of the query image with feature points in each key image in the candidate key image set, and filtering out the feature points of the query image at optimal and suboptimal ratios to obtain filtered feature points of the query image;

[0012] determining the position and posture when the query image was taken according to the feature points of the filtered query image;

[0013] The indoor map includes key image information for constructing the indoor map, feature point information in the key image for constructing the indoor map, and three-dimensional map point information for constructing the indoor map.

[0014] In one embodiment, querying an indoor map according to the descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image includes:

[0015] All key images in the indoor map that have the same feature points as the query image are obtained.

[0016] In one embodiment, querying an indoor map according to the descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image includes:

[0017] A specific proportion of key images of all key images of the indoor map having the same feature points as the query image is obtained in descending order of the number of feature points that are the same as the query image.

[0018] In one embodiment, querying an indoor map according to the descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image includes:

[0019] In the key images of the indoor map, a specific number of key image groups adjacent to each key image in the key images of the specific ratio are obtained in descending order of the number of common view feature points contained therein.

[0020] In one embodiment, querying an indoor map according to the descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image includes:

[0021] A key image group of a specific ratio is obtained by calculating the sum of the number of identical feature points between each key image in the key image group of each key image in the key images of the specific ratio and the query image from large to small, and all key images in the key image group of the specific ratio are used as the candidate key image set.

[0022] In one embodiment, filtering out the feature points of the query image at an optimal and suboptimal ratio to obtain the filtered feature points of the query image includes:

[0023] When comparing a feature point of the query image with a feature point in a key image in the candidate key image set, if the ratio of the similarity between the feature point of the query image and the two feature points in the key image in the candidate key image set is greater than or equal to a specific ratio, the feature point of the query image is screened out.

[0024] In one embodiment, filtering out the feature points of the query image at an optimal and suboptimal ratio to obtain the filtered feature points of the query image includes:

[0025] Only a feature point of the query image that is screened out in the comparison between a feature point of the query image and a feature point in each key image in the candidate key image set will be finally screened out.

[0026] According to one aspect of the present disclosure, a visual positioning device for indoor navigation is provided, comprising:

[0027] an acquisition module configured to acquire a query image of the room, acquire feature points of the query image, and acquire descriptors of the feature points of the query image;

[0028] a query module configured to query an indoor map according to a descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image;

[0029] a comparison and screening module configured to compare each feature point of the query image with feature points in each key image in the candidate key image set, and screen out the feature points of the query image at an optimal and suboptimal ratio to obtain feature points of the filtered query image;

[0030] a positioning module, for determining the position and posture when the query image was taken based on the feature points of the filtered query image;

[0031] The indoor map includes key image information for constructing the indoor map, feature point information in the key image for constructing the indoor map, and three-dimensional map point information for constructing the indoor map.

[0032] According to one aspect of the present disclosure, there is provided an electronic device, including:

[0033] one or more processors;

[0034] The storage device is configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method as described in any one of the above embodiments.

[0035] According to one aspect of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of the above embodiments is implemented.

[0036] In the technical solutions provided in some embodiments of the present disclosure, low-cost visual positioning technology is used to improve the user's positioning accuracy.

[0037] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The following drawings describe certain illustrative embodiments of the present invention, wherein like reference numerals represent like elements. These described embodiments are intended to be exemplary embodiments of the present disclosure and are not intended to be limiting in any way.

[0039] Figure 1 A schematic diagram showing an exemplary system architecture to which the visual positioning method for indoor navigation or the visual positioning device for indoor navigation according to an embodiment of the present disclosure can be applied;

[0040] Figure 2 A schematic diagram showing the structure of a computer system suitable for implementing the electronic device of the embodiment of the present disclosure is shown;

[0041] Figure 3 Schematically illustrates a schematic diagram of forming three-dimensional points when constructing an indoor map in visual positioning for indoor navigation according to an embodiment of the present disclosure;

[0042] Figure 4 The following schematically shows a flow chart of a visual positioning method for indoor navigation according to an embodiment of the present disclosure;

[0043] Figure 5 Schematically shows a schematic diagram of room pose estimation in visual positioning for indoor navigation according to an embodiment of the present disclosure;

[0044] Figure 6 A block diagram of a visual positioning device for indoor navigation according to an embodiment of the present disclosure is schematically shown;

[0045] Figure 7 Schematically shows a block diagram of a visual positioning device for indoor navigation according to another embodiment of the present invention;

[0046] Figure 8 The following schematically shows a block diagram of a visual positioning device for indoor navigation according to another embodiment of the present invention. DETAILED DESCRIPTION

[0047] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that this disclosure will be thorough and complete and will fully convey the concepts of the example embodiments to those skilled in the art.

[0048] In addition, the described features, structures or characteristics may be combined in any suitable manner in one or more embodiments. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be employed. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid obscuring various aspects of the present disclosure.

[0049] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0050] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0051] Figure 1 A schematic diagram of an exemplary system architecture 100 is shown, to which the visual positioning method for indoor navigation or the visual positioning device for indoor navigation according to an embodiment of the present disclosure can be applied.

[0052] like Figure 1 As shown, system architecture 100 may include one or more terminal devices 101, 102, 103, a network 104, and a server 105. Network 104 is a medium for providing a communication link between terminal devices 101, 102, 103 and server 105. Network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0053] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as needed. For example, the server 105 may be a server cluster consisting of multiple servers.

[0054] Staff or customers can use terminal devices 101, 102, 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, 103 can be various electronic devices with display screens, including but not limited to smartphones, tablet computers, laptop computers, desktop computers, digital movie projectors, etc.

[0055] Server 105 can be a server that provides various services. For example, a staff member uses terminal device 103 (which can also be terminal device 101 or 102) to send a visual positioning request for indoor navigation to server 105. Server 105 can obtain an indoor query image; obtain feature points of the query image; obtain descriptors of the feature points of the query image; query an indoor map based on the descriptors of the query image to obtain a set of candidate key images in the indoor map of the query image; compare each feature point of the query image with the feature points in each key image in the candidate key image set, and filter the feature points of the query image at optimal and suboptimal ratios to obtain feature points of the filtered query image; determine the position and posture when the query image was taken based on the feature points of the filtered query image; wherein the indoor map includes key image information for constructing the indoor map, including feature point information in the key image for constructing the indoor map, and also includes three-dimensional map point information for constructing the indoor map. The server 105 can send the position and posture when the query image is taken to the terminal device 103 to display the position and posture when the query image is taken on the terminal device 103, so that the staff can perform indoor positioning or navigation based on the content displayed on the terminal device 103.

[0056] For another example, the terminal device 103 (which may also be the terminal device 101 or 102) may be a smart TV, a VR (Virtual Reality) / AR (Augmented Reality) helmet display, or a mobile terminal such as a smart phone or tablet computer on which a navigation, online car-hailing, instant messaging, or video application (APP) is installed. The staff may send a visual positioning request for indoor navigation to the server 105 through the smart TV, VR / AR helmet display, or the navigation, online car-hailing, instant messaging, or video APP. The server 105 may obtain the position and posture when the query image was taken based on the visual positioning request for indoor navigation, and return the position and posture when the query image was taken to the smart TV, VR / AR helmet display, or the navigation, online car-hailing, instant messaging, or video APP, and then display the position and posture when the query image was taken through the smart TV, VR / AR helmet display, or the navigation, online car-hailing, instant messaging, or video APP.

[0057] Figure 2 A schematic structural diagram of a computer system suitable for implementing the electronic device according to the embodiment of the present disclosure is shown.

[0058] It should be noted that Figure 2 The computer system 200 of the electronic device shown is only an example and should not limit the functions and scope of use of the embodiments of the present disclosure.

[0059] like Figure 2 As shown, computer system 200 includes a central processing unit (CPU) 201, which can perform various appropriate actions and processes according to programs stored in read-only memory (ROM) 202 or programs loaded from storage unit 208 into random access memory (RAM) 203. Various programs and data required for system operation are also stored in RAM 203. CPU 201, ROM 202, and RAM 203 are connected to each other via bus 204. Input / output (I / O) interface 205 is also connected to bus 204.

[0060] The following components are connected to the I / O interface 205: an input section 206 including a keyboard, a mouse, and the like; an output section 207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 208 including a hard disk; and a communication section 209 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 209 performs communication processing via a network such as the Internet. A drive 210 is also connected to the I / O interface 205 as needed. A removable medium 211, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 210 as needed, so that a computer program read therefrom can be installed into the storage section 208 as needed.

[0061] In particular, according to an embodiment of the present disclosure, the process described below with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable storage medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 209, and / or installed from a removable medium 211. When the computer program is executed by the central processing unit (CPU) 201, the various functions defined in the method and / or apparatus of the present application are performed.

[0062] It should be noted that the computer-readable storage medium described in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable storage medium other than a computer-readable storage medium that can send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF (Radio Frequency), etc., or any suitable combination thereof.

[0063] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the methods, devices and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions selected in the box can also occur in an order different from the order selected in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0064] The modules and / or units and / or sub-units described in the embodiments of the present disclosure may be implemented in software or hardware, and the modules and / or units and / or sub-units may also be provided in a processor. The names of these modules and / or units and / or sub-units do not, in certain circumstances, limit the modules and / or units and / or sub-units themselves.

[0065] As another aspect, the present application also provides a computer-readable storage medium, which may be included in the electronic device described in the above embodiment; or it may exist independently without being assembled into the electronic device. The above computer-readable storage medium carries one or more programs, and when the above one or more programs are executed by an electronic device, the electronic device implements the method described in the following embodiment. For example, the electronic device may implement the following embodiment. Figure 4 Each step.

[0066] In related technologies, for example, machine learning methods, deep learning methods, etc. can be used for visual positioning of indoor navigation, and different methods have different scopes of application.

[0067] Figure 3 The diagram schematically shows the formation of three-dimensional points when constructing an indoor map in visual positioning of indoor navigation according to an embodiment of the present disclosure.

[0068] Indoor map construction

[0069] 3D reconstruction solves the problem of indoor map construction. 3D reconstruction generally requires the use of Structure from Motion (SfM), which is defined as "enabling a robot to construct an environment and estimate its own motion in an unknown environment." In layman's terms, it uses images or videos captured by a camera to restore the 3D structure of the entire scene. The input of the system is multiple images or video streams, and the output is the 3D structure of the scene and the pose of each image. The camera pose has 6 degrees of freedom, 3 degrees of freedom for position, and 3 degrees of freedom for attitude (camera orientation). Position can be understood as a point in 3D space, and attitude is the orientation of the camera.

[0070] The technical framework of SfM consists of four main steps: ① data acquisition; ② feature extraction; ③ data association; ④ structure recovery. When recovering the three-dimensional structure of a scene, not all pixels in the image are used, but only stable and significant points, that is, feature points, are extracted from the image; data association is to determine which feature points in two images with a common view area are corresponding; subsequently, geometric methods and optimization techniques (Bundle Adjustment) will be used to restore the points in these images to three-dimensional points in space. The entire problem is eventually converted into a large-scale nonlinear optimization problem to be solved. The optimization goal is to minimize the difference between the position of the reconstructed three-dimensional point reprojected into the image and the position of the observation point in the image, that is, to minimize the reprojection error.

[0071] For indoor positioning scenarios, this application adopts a monocular vision mapping solution: a camera is used to capture rich image samples in indoor scenes, and then the SfM technology is used to reconstruct the three-dimensional structure. During the SfM calculation process, the collected image samples are divided into a few key frames (key images) and a majority of non-key frames according to changes in viewing angle and moving distance. These key frames have obvious observation parallax. Due to their small number, the maps constructed by the feature points they contain are small and precise. There are many feature and descriptor algorithms, and the more mature ones are SIFT (Scale-invariant feature transform), SURF (Speeded Up Robust Feature), ORB (Oriented FAST and Rotated BRIEF, an algorithm for fast feature point extraction and description), etc. Such as Figure 3 As shown in Figure 1, the 3D map points generated during the SfM calculation process form a large and complex feature matching and association between keyframes. As a result, each keyframe has several "neighboring" keyframes with different numbers of common viewpoints between them.

[0072] The 3D reconstruction algorithm is computationally intensive and slow, so it can be completed offline. When the map is built, it is saved as a single file. The map elements include:

[0073] ① 3D map point coordinates and ID (identity) identification;

[0074] ② Feature point coordinates, descriptors, ID identifiers, corresponding 3D map point ID identifiers, and key frame ID identifiers;

[0075] ③The position, posture, ID of the key frame, the set of feature point IDs included, and the neighbor key frames and IDs sorted by weight (number of common viewpoints).

[0076] The key image information of the indoor map may be ③, the feature point information in the key image of the indoor map may be ②, and the three-dimensional map point information of the indoor map may be ①.

[0077]

[0078] In formula (1):

[0079] C1~C N Represents N camera poses;

[0080] X1~X N Represents N three-dimensional space points;

[0081] π(X i ,C j ) is the projection transformation formula, the spatial point X i In camera C j Projection coordinates;

[0082] x ij Represents a spatial point X i In camera C j The actual two-dimensional observation coordinates in ;

[0083] ||e|| means finding the second norm of the vector, that is, the Euclidean distance;

[0084] The square of the Euclidean distance is used as the optimization target to facilitate the solution of mathematical aspects such as derivatives.

[0085] Figure 4 The flowchart of the visual positioning method for indoor navigation according to an embodiment of the present disclosure is schematically shown. The method steps of the embodiment of the present disclosure can be executed by the terminal device, or by the server, or by the terminal device and the server interactively. For example, the method steps can be executed by the terminal device or the server interactively. Figure 1 The server 105 in the embodiment executes the above operations, but the present disclosure is not limited to this.

[0086] In step S410, an indoor query image is acquired.

[0087] In this step, the terminal device or server obtains the indoor query image. In practical applications, this can be achieved by the user using a mobile terminal or other device to capture the environment image.

[0088] In the embodiments of the present disclosure, terminal devices may be implemented in various forms. For example, the terminals described in the present disclosure may include mobile terminals such as mobile phones, tablet computers, laptop computers, PDAs, portable media players (PMPs), visual positioning devices for indoor navigation, wearable devices, smart bracelets, pedometers, robots, and driverless cars, as well as fixed terminals such as digital TVs and desktop computers.

[0089] In step S420, feature points of the query image are obtained.

[0090] In this step, the terminal device or the server obtains the feature points of the query image.

[0091] In step S430 , descriptors of feature points of the query image are obtained.

[0092] In this step, the terminal device or the server obtains the descriptors of the feature points of the query image. In one embodiment, steps 420 and 430 can be performed simultaneously.

[0093] In step S440, an indoor map is queried according to the descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image.

[0094] In this step, the terminal device or server queries the indoor map based on the descriptor of the query image to obtain a set of candidate key images in the indoor map for the query image. In one embodiment, the indoor map includes key image information for constructing the indoor map, including feature point information within the key images for constructing the indoor map, and also includes three-dimensional map point information for constructing the indoor map. In one embodiment, all key images in the indoor map that share feature points with the query image are obtained. In one embodiment, a specific proportion of key images are obtained from all key images in the indoor map that share feature points with the query image, in descending order of the number of feature points they share with the query image. For example, the top 20% of key images that share feature points with the query image are selected. In one embodiment, a specific number of key image groups adjacent to each key image in the specific proportion of key images are obtained, in descending order of the number of common view feature points they contain. The specific number of key image groups is, for example, five. In one embodiment, a key image group of a specific ratio is obtained by taking the sum of the number of identical feature points between each key image in the key image group of each key image in the key images of the specific ratio and the query image from large to small, and all key images in the key image group of a specific ratio are used as the candidate key image set. The key image group of a specific ratio is, for example, the top 20% key image groups in terms of the sum of the number of identical feature points with the query image from large to small.

[0095] In step S450, each feature point of the query image is compared with the feature points of each key image in the candidate key image set, and the feature points of the query image are screened out at optimal and suboptimal ratios to obtain the filtered feature points of the query image.

[0096] In this step, the terminal device or server compares each feature point of the query image with the feature points in each key image in the candidate key image set, and screens the feature points of the query image at the optimal and suboptimal ratios to obtain the feature points of the query image after screening. In one embodiment, when a feature point of the query image is compared with a feature point in a key image in the candidate key image set, if the ratio of the similarity between the feature point of the query image and the two feature points in the key image in the candidate key image set is greater than or equal to a specific ratio, the feature point of the query image is screened out. The specific ratio is, for example, a 4:1 ratio of the similarity between the two feature points. In one embodiment, only the feature point of the query image that is screened out in the comparison between the feature point of the query image and the feature points in each key image in the candidate key image set will be finally screened out.

[0097] In step S460 , the position and posture when the query image was taken are determined based on the feature points of the filtered query image.

[0098] In this step, the terminal device or server determines the position and pose of the query image when it was captured based on the feature points of the filtered query image. In one embodiment, the position and pose of the query image are calculated using a RANSAC (RANdom Sampling Consensus) + PnP (Perspective-n-Point) algorithm based on the feature points of the filtered query image.

[0099] The visual positioning method for indoor navigation of the present application can obtain an indoor query image and query the indoor query image in an indoor map to determine the position and posture when the query image was taken.

[0100] The visual positioning method for indoor navigation of the present application determines the position and posture when the query image is captured and can be combined with augmented reality technology to display guidance information on the user's mobile phone, providing the user with a what-you-see-is-what-you-get interactive experience.

[0101] Figure 5 The figure schematically shows a schematic diagram of the pose estimation of a room in the visual positioning of indoor navigation according to an embodiment of the present disclosure.

[0102] Monocular vision positioning

[0103] In the use scenario of AR (augmented reality) navigation, the purpose of positioning is to determine the shooting position and direction of the camera. After obtaining a query image, the image is subjected to feature and descriptor extraction. The method used in this application is consistent with the algorithm used in the indoor map construction stage. Based on the large number of 2D key points and descriptors obtained in the image, if the image is converted into a bag-of-words vector, the matching frame (matching image) in the key frame that is most likely to be relocated is retrieved. Although this approach is simple, due to the instability of the score of a single frame, this paper adopts a more robust candidate key frame retrieval scheme:

[0104] Search the indoor map for keyframes with the same descriptors as the query graph, along with their weights (number of common viewpoints); their set is denoted as:

[0105] S0={(KeyFrameID i ,CommonDescNum i )|[i]hasCommonDescWith[F]}(2)

[0106] In formula (2), the │ after represents the condition (having a common descriptor with the query image), and the │ before represents the key frame's identity and the number of the same descriptors;

[0107] The threshold is 0.8 times the number of the largest identical descriptors:

[0108] NumThresh=max({x.KeyFrameID|x∈S0})*0.8(3)

[0109] In formula (3), the │ after represents the condition (in S0), and the │ before represents the identity of the key frame;

[0110] According to the threshold, the key frame set S1 greater than or equal to 0.8 times the threshold is filtered from S0 again:

[0111] S1={x.KeyFrameID|x.Num>NumThresh, x∈S0}(4)

[0112] In formula (4), the │ after represents the condition (greater than 0.8 times the threshold), and the │ before represents the identity of the key frame;

[0113] In order to fully explore and utilize the contextual information relationship in the map, for each key frame in the set S1, the M (actually set to 5) key frames with the largest weight connected in the co-visual map are selected, and the matching score with the query frame (query image) is calculated respectively:

[0114]

[0115] G={g y |y∈S1}(6)

[0116]

[0117] Formulas (5)-(7) represent the process of obtaining the scores of the M (actually set to 5) key frames with the largest connected weights;

[0118] Take it as a whole and get the overall matching score. Finally, return the frame with a group score greater than a certain value (0.8 times the maximum value) and the highest score in the group:

[0119] ScoreThresh=max({AccScore y |y∈S1})*0.8(8)

[0120] S2={g y .KeyFrameID|AccScore y >ScoreThresh,y∈S1}(9)

[0121] Formulas (8)-(9) represent obtaining a set of key frames S2 with scores greater than a certain value (0.8 times the maximum value);

[0122] After finding the candidate keyframe set S2, perform a feature match between each candidate frame and the query frame, and filter out the feature points using the optimal and suboptimal ratios. The remaining 2D feature points after filtering are matched with the 3D points in the model, and finally RANSAC+PnP is used to solve and calculate the position and posture [R|t] of the query graph, for example Figure 5 shown.

[0123] "6DoF" stands for "6 degrees of freedom," or six degrees of freedom in Chinese. Six degrees of freedom refers to the six basic motions derived from 3D movement. Generally speaking, 3D movement can be broadly divided into two categories: translation (i.e., parallel linear movement) and rotation. However, the vectors in 3D space are the X, Y, and Z axes, and each axis has translation and rotation. Therefore, there are a total of six basic motions in 3D space, the so-called 6DOF.

[0124] Figure 6 The block diagram of the visual positioning device for indoor navigation according to one embodiment of the present disclosure is schematically shown. The visual positioning device 600 for indoor navigation provided by the embodiment of the present disclosure can be set on the terminal device, can also be set on the server side, or can be set partially on the terminal device and partially on the server side. For example, it can be set Figure 1 The server 105 in the embodiment of the present invention is provided, but the present disclosure is not limited thereto.

[0125] The visual positioning device 600 for indoor navigation provided by the embodiment of the present disclosure may include an acquisition module 610 , a query module 620 , a comparison and screening module 630 , and a positioning module 640 .

[0126] Among them, the acquisition module is configured to acquire an indoor query image, acquire feature points of the query image, and acquire descriptors of the feature points of the query image; the query module is configured to query the indoor map based on the descriptors of the query image to obtain a set of candidate key images in the indoor map of the query image; the comparison and screening module is configured to compare each feature point of the query image with the feature points in each key image in the candidate key image set, and screen the feature points of the query image at optimal and suboptimal ratios to obtain the feature points of the filtered query image; the positioning module determines the position and posture when the query image is taken based on the feature points of the filtered query image; wherein the indoor map includes key image information for constructing the indoor map, feature point information in the key image for constructing the indoor map, and three-dimensional map point information for constructing the indoor map.

[0127] According to the embodiment of the present disclosure, the visual positioning device 600 for indoor navigation can be used to implement Figure 4 The embodiment describes a visual positioning method for indoor navigation.

[0128] Figure 7 A block diagram of a visual positioning device 700 for indoor navigation according to another embodiment of the present invention is schematically shown.

[0129] like Figure 7 As shown, except Figure 6 In addition to the acquisition module 610 , query module 620 , comparison and screening module 630 , and positioning module 640 described in the embodiment, the visual positioning device 700 for indoor navigation further includes a display module 710 .

[0130] Specifically, the display module 710 is configured to display the position and posture when the query image is captured, determined based on the feature points of the filtered query image.

[0131] In the visual positioning device 700 for indoor navigation, the display module 710 can be used to intuitively display the position and posture when the query image was taken based on the feature points of the filtered query image.

[0132] Figure 8 A block diagram of a visual positioning device 800 for indoor navigation according to another embodiment of the present invention is schematically shown.

[0133] like Figure 8 As shown, except Figure 6 In addition to the acquisition module 610 , the query module 620 , the comparison and screening module 630 , and the positioning module 640 described in the embodiment, the visual positioning device 800 for indoor navigation further includes a storage module 810 .

[0134] Specifically, the storage module 810 is used to store various data called by the user during use in the database for repeated query and inspection.

[0135] It is understandable that the acquisition module 610, the query module 620, the comparison and screening module 630, the positioning module 640, the display module 710, and the storage module 810 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the acquisition module 610, the query module 620, the comparison and screening module 630, the positioning module 640, the display module 710, and the storage module 810 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented in hardware or firmware in any other reasonable way of integrating or packaging the circuit, or in an appropriate combination of software, hardware, and firmware. Alternatively, at least one of the acquisition module 610, query module 620, comparison and screening module 630, positioning module 640, display module 710, and storage module 810 can be at least partially implemented as a computer program module, and when the program is run by a computer, the functions of the corresponding module can be executed.

[0136] Since the various modules of the visual positioning device for indoor navigation of the exemplary embodiment of the present invention can be used to implement the above Figure 4 The steps of an example implementation of the visual positioning method for indoor navigation are described, so for details not disclosed in the implementation of the device of the present invention, please refer to the implementation of the visual positioning method for indoor navigation of the present invention described above.

[0137] The specific implementation of each module, unit and sub-unit in the visual positioning device for indoor navigation provided by the embodiment of the present disclosure can refer to the contents of the visual positioning method for indoor navigation mentioned above, and will not be repeated here.

[0138] It should be noted that although several modules, units, and sub-units of the device for action execution are mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules, units, and sub-units described above can be embodied in one module, unit, and sub-unit. Conversely, the features and functions of one module, unit, and sub-unit described above can be further divided into multiple modules, units, and sub-units to be embodied.

[0139] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) to execute the method according to the embodiments of the present disclosure.

[0140] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and embodiments are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the following claims.

[0141] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A visual positioning method for indoor navigation, characterized in that: include: Get an indoor query image; Acquire feature points of the query image; Obtaining descriptors of feature points of the query image; querying an indoor map according to the descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image; Comparing each feature point of the query image with feature points in each key image in the candidate key image set, and filtering out the feature points of the query image at optimal and suboptimal ratios to obtain filtered feature points of the query image; determining the position and posture when the query image was taken according to the feature points of the filtered query image; The indoor map includes key image information for constructing the indoor map, feature point information in the key image for constructing the indoor map, and three-dimensional map point information for constructing the indoor map; Among them, querying the indoor map according to the descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image includes: searching the indoor map for key images containing the same descriptors as those in the query image, and weights, to determine a first key image set; using 0.8 times the maximum number of identical descriptors as a threshold; based on the threshold, again screening from the first key image set to obtain a second key image set greater than or equal to 0.8 times the threshold; for each key image in the second key image set, respectively, according to the M key images with the largest weights connected in the co-visual map, and respectively calculating the matching score with the query image; and determining a set of key images with scores greater than 0.8 times the maximum value as the candidate key image set.

2. The method according to claim 1, characterized in that Querying an indoor map according to the descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image includes: All key images in the indoor map that have the same feature points as the query image are obtained.

3. The method according to claim 2, characterized in that Querying an indoor map according to the descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image includes: Obtaining, in descending order of the number of feature points common to the query image, a specific proportion of key images from all key images of the indoor map having the same feature points as the query image; The specific proportion is the first 20%.

4. The method according to claim 3, characterized in that Querying an indoor map according to the descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image includes: In the key images of the indoor map, a specific number of key image groups adjacent to each key image in the key images of the specific ratio are obtained in descending order of the number of common view feature points contained therein; Among them, the specific number is 5.

5. The method according to claim 4, characterized in that Querying an indoor map according to the descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image includes: A key image group of a specific ratio is obtained by calculating the sum of the number of identical feature points between each key image in the key image group of each key image in the key images of the specific ratio and the query image from large to small, and all key images in the key image group of the specific ratio are used as the candidate key image set.

6. The method according to claim 1, characterized in that Screening the feature points of the query image at the optimal and suboptimal ratios to obtain the filtered feature points of the query image includes: When comparing a feature point of the query image with a feature point in a key image in the candidate key image set, if a ratio of similarities between the feature point of the query image and two feature points in the key image in the candidate key image set is greater than or equal to a specific ratio, then filtering out the feature point of the query image; Wherein, the specific ratio is 4:

1.

7. The method according to claim 6, characterized in that Screening the feature points of the query image at the optimal and suboptimal ratios to obtain the filtered feature points of the query image includes: Only a feature point of the query image that is screened out in the comparison between a feature point of the query image and a feature point in each key image in the candidate key image set will be finally screened out.

8. A visual positioning device for indoor navigation, characterized in that: include: an acquisition module configured to acquire a query image of the room, acquire feature points of the query image, and acquire descriptors of the feature points of the query image; a query module configured to query an indoor map according to a descriptor of the query image to obtain a set of candidate key images in the indoor map of the query image; a comparison and screening module configured to compare each feature point of the query image with feature points in each key image in the candidate key image set, and screen out the feature points of the query image at an optimal and suboptimal ratio to obtain feature points of the filtered query image; a positioning module, for determining the position and posture when the query image was taken based on the feature points of the filtered query image; The indoor map includes key image information for constructing the indoor map, feature point information in the key image for constructing the indoor map, and three-dimensional map point information for constructing the indoor map; The query module is further configured to search the indoor map for key images containing the same descriptors as those in the query graph, as well as weights, to determine a first key image set; using 0.8 times the maximum number of identical descriptors as a threshold; and again screening the first key image set based on the threshold to obtain a second key image set greater than or equal to 0.8 times the threshold; for each key image in the second key image set, respectively obtain the matching scores with the query image according to the M key images with the largest weights connected in the co-visual graph; and determine the key image set with a score greater than 0.8 times the maximum value as the candidate key image set.

9. An electronic device, characterized in that: include: one or more processors; A storage device configured to store one or more programs, which, when executed by the one or more processors, causes the one or more processors to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images