Visual positioning method and system based on indoor building frame constraints

Abstract

This invention provides a visual positioning method and system based on indoor building frame constraints. The method includes acquiring an image to be processed, which includes a first camera image and a second camera image; extracting the building frame from the image to obtain a semantic planar map; simultaneously extracting feature points and feature descriptors from the image to obtain feature points and their corresponding feature descriptors; constructing a feature point matching model, and training the feature point matching model using the semantic planar map corresponding to the first camera image, each feature point and its corresponding feature descriptor, and the semantic planar map corresponding to the second camera image, each feature point and its corresponding feature descriptor, as a dataset; and matching the camera images to be matched based on the trained feature point matching model to obtain the corresponding points of the camera images to be matched, so as to perform camera pose estimation based on the corresponding points of the camera images to be matched; thereby reducing interference from dynamic objects and improving the positioning effect.

Description

Technical Field

[0001] This invention relates to the field of positioning technology, and in particular to a visual positioning method and system based on indoor building frame constraints. Background Technology

[0002] In related technologies, with the rapid development of intelligent robots, the intelligent service industry is booming. Robots can replace humans in the service field to complete specific tasks, such as unmanned delivery vehicles and sweeping robots. These intelligent robots need to perceive the surrounding environment and realize autonomous navigation and positioning functions in order to complete the tasks. At present, indoor mobile robots are emerging in shopping malls, security and other fields, and indoor visual positioning has also become a popular research direction.

[0003] Visual localization, also known as camera localization, is a problem of solving the 6DoF pose of a camera in a world coordinate system. Depending on the task, it can be divided into solving relative camera pose and absolute camera pose. This invention mainly studies the relative pose estimation method. Due to the low cost and easy installation of cameras, visual localization often uses cameras as visual sensors. The mainstream method of relative pose estimation is to extract obvious and repeatable feature points from two frames of images, find the initial correspondence between feature points in the two frames through feature matching, eliminate abnormal matches, and finally estimate the camera pose. However, current relative pose methods based on feature matching assume that objects in the environment are stationary and only the camera is moving. But in real-world scenarios, there are a lot of moving objects, such as repeatedly opened and closed doors and windows, frequently moved tables and chairs, and people walking around. These dynamic objects can bring incorrect observation data to the system and reduce the robustness of the system. Although the RANSAC algorithm can filter out some abnormal matches, when dynamic objects occupy most of the image area, brute-force removal of these objects may destroy the quantity and quality of feature matching, especially when other static areas in the scene cannot provide enough features, which will still seriously affect the localization effect. Summary of the Invention

[0004] The present invention aims to at least partially solve one of the technical problems in the aforementioned technologies. Therefore, one objective of the present invention is to propose a visual positioning method based on indoor building frame constraints, which utilizes the invariance of the building frame to guide the network to focus on feature points on static objects, reducing interference from dynamic objects, thereby improving the positioning effect.

[0005] The second objective of this invention is to propose a visual positioning system based on the constraints of an indoor building frame.

[0006] To achieve the above objectives, a first aspect of the present invention proposes a visual localization method based on indoor building frame constraints, comprising: acquiring an image to be processed, wherein the image to be processed includes a first camera image and a second camera image; then, extracting the building frame from the image to be processed to obtain a semantic plane map of the same size as the image to be processed; simultaneously extracting feature points and feature descriptors from the image to be processed to obtain each feature point in the image to be processed and its corresponding feature descriptor; constructing a feature point matching model, and training the feature point matching model using the semantic plane map corresponding to the first camera image, each feature point and its corresponding feature descriptor, and the semantic plane map corresponding to the second camera image, each feature point and its corresponding feature descriptor, as a dataset; and matching the camera images to be matched based on the trained feature point matching model to obtain the corresponding points of the camera images to be matched, so as to perform camera pose estimation based on the corresponding points of the camera images to be matched.

[0007] According to the visual localization method based on indoor building frame constraints of the present invention, firstly, an image to be processed is acquired, wherein the image to be processed includes a first camera image and a second camera image; then, the building frame is extracted from the image to be processed to obtain a semantic plane map of the same size as the image to be processed; simultaneously, feature points and feature descriptors are extracted from the image to be processed to obtain each feature point in the image to be processed and its corresponding feature descriptor; next, a feature point matching model is constructed, and the semantic plane map corresponding to the first camera image, each feature point and its corresponding feature descriptor, and the semantic plane map corresponding to the second camera image, each feature point and its corresponding feature descriptor are used as a dataset to train the feature point matching model; finally, the trained feature point matching model is used to match the camera images to be matched to obtain the corresponding points of the camera images to be matched, so as to perform camera pose estimation based on the corresponding points of the camera images to be matched; thereby, the invariance of the building frame can be used to guide the network to focus on feature points on static objects, reduce the interference of dynamic objects, and thus improve the localization effect.

[0008] In addition, the visual positioning method based on indoor building frame constraints proposed in the above embodiments of the present invention may also have the following additional technical features:

[0009] Preferably, the image to be processed is subjected to architectural frame extraction to obtain a semantic plane map with the same size as the image to be processed, including: extracting architectural frames from the image to be processed based on an improved ResNet101 network, so as to segment the image to be processed into a semantic plane map composed of multiple small semantic planes according to different interior surfaces of buildings.

[0010] Preferably, feature point and feature descriptor extraction is performed on the image to be processed to obtain each feature point in the image to be processed and its corresponding feature descriptor, including: using the feature point decoder in the trained SuperPoint model to detect the location of the feature points, and using sub-pixel convolution to reduce the computational load in order to extract the feature points; and using the feature descriptor decoder in the trained SuperPoint model to generate the feature descriptor corresponding to each feature point in the image to be processed.

[0011] Preferably, training the feature point matching model includes: using positional encoding to map the positional information of each feature point into a high-dimensional vector, so as to fuse the positional information with its corresponding feature descriptor; improving the receptive field of the fused feature descriptor through an attention mechanism, so as to enhance the feature communication between the first camera image and the second camera image; constructing a similarity matrix of feature descriptors to obtain the corresponding point set of the first camera image and the second camera image; obtaining the confidence of the semantic plane map corresponding to the first camera image and the second camera image based on the corresponding point set; constructing a first loss function based on the true value of each pair of corresponding points, and constructing a second loss function based on the corresponding point set of the first camera image and the second camera image and the confidence of the semantic plane map, so as to obtain the final loss function based on the first loss function and the second loss function.

[0012] Preferably, after obtaining the corresponding points of the camera image to be matched, the data of abnormal matches are filtered out by the RANSAC algorithm.

[0013] Preferably, the camera pose estimation based on the corresponding points of the camera image to be matched includes: using the OpenCV library to solve for the essential matrix of the corresponding points of the camera image to be matched after filtering out abnormal matching data, so as to estimate the rotation and translation of the camera image to be matched.

[0014] To achieve the above objectives, a second aspect of the present invention provides a visual positioning system based on indoor building frame constraints, comprising: an acquisition module for acquiring an image to be processed, wherein the image to be processed includes a first camera image and a second camera image; a first extraction module for extracting the building frame from the image to be processed to obtain a semantic plane map of the same size as the image to be processed; a second extraction module for extracting feature points and feature descriptors from the image to be processed to obtain each feature point and its corresponding feature descriptor in the image to be processed; a training module for constructing a feature point matching model and training the feature point matching model using the semantic plane map corresponding to the first camera image, each feature point and its corresponding feature descriptor, and the semantic plane map corresponding to the second camera image, each feature point and its corresponding feature descriptor as a dataset; and an estimation and localization module for matching the camera images to be matched based on the trained feature point matching model to obtain the corresponding points of the camera images to be matched, so as to perform camera pose estimation based on the corresponding points of the camera images to be matched.

[0015] The visual positioning system based on indoor building frame constraints provided by the present invention can utilize the invariance of the building frame to guide the network to focus on feature points on static objects, reduce interference from dynamic objects, and thus improve the positioning effect.

[0016] In addition, the visual positioning system based on indoor building frame constraints proposed in the above embodiments of the present invention may also have the following additional technical features:

[0017] Preferably, the first extraction module is further configured to extract architectural frames from the image to be processed based on an improved ResNet101 network, so as to segment the image to be processed into a semantic plane map composed of multiple small semantic planes according to different interior architectural surfaces.

[0018] Preferably, the second extraction module is further configured to: use the feature point decoder in the trained SuperPoint model to detect the location of feature points, and use sub-pixel convolution to reduce the computational load in order to extract feature points; and use the feature descriptor decoder in the trained SuperPoint model to generate a feature descriptor corresponding to each feature point in the image to be processed.

[0019] Preferably, the training module is further configured to: map the positional information of each feature point into a high-dimensional vector using positional encoding, so as to fuse the positional information with its corresponding feature descriptor; enhance the receptive field of the fused feature descriptor through an attention mechanism, so as to enhance the feature communication between the first camera image and the second camera image; construct a similarity matrix of feature descriptors to obtain the corresponding point set of the first camera image and the second camera image; obtain the confidence of the semantic plane map corresponding to the first camera image and the second camera image based on the corresponding point set; construct a first loss function based on the true value of each pair of corresponding points, and construct a second loss function based on the corresponding point set of the first camera image and the second camera image and the confidence of the semantic plane map, so as to obtain the final loss function based on the first loss function and the second loss function. Attached Figure Description

[0020] Figure 1 This is a flowchart illustrating a visual positioning method based on indoor building frame constraints according to an embodiment of the present invention.

[0021] Figure 2 This is a schematic diagram of extracting the building frame according to an embodiment of the present invention;

[0022] Figure 3 This is a schematic diagram illustrating the extraction of feature points and feature descriptors according to an embodiment of the present invention;

[0023] Figure 4 This is a schematic block diagram illustrating feature matching according to an embodiment of the present invention;

[0024] Figure 5 This is a block diagram of a visual positioning system based on indoor building frame constraints according to an embodiment of the present invention. Detailed Implementation

[0025] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0026] To better understand the above technical solutions, exemplary embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although exemplary embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided to enable a more thorough understanding of the present invention and to fully convey the scope of the invention to those skilled in the art.

[0027] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0028] This invention proposes a visual localization method based on indoor building frame constraints for indoor visual localization tasks in dynamic scenes. Since the indoor building frame is stationary, it provides a reference for whether objects move in two frames. By adding building frame constraints, the network can better focus on feature points on static objects during training, thereby reducing interference from dynamic objects.

[0029] Figure 1 This is a flowchart illustrating a visual positioning method based on indoor building frame constraints according to an embodiment of the present invention. Figure 1 As shown, the visual positioning method based on indoor building frame constraints according to an embodiment of the present invention includes the following steps:

[0030] S101, acquire the image to be processed, wherein the image to be processed includes the first camera image and the second camera image.

[0031] In other words, two frames of images are continuously acquired by the camera, which are the first camera image and the second camera image.

[0032] S102, extract the architectural framework from the image to be processed to obtain a semantic planar graph with the same size as the image to be processed.

[0033] As one embodiment, architectural framework extraction is performed on the image to be processed to obtain a semantic plane map with the same size as the image to be processed. This includes: extracting architectural frameworks from the image to be processed based on an improved ResNet101 network, so as to segment the image to be processed into a semantic plane map composed of multiple small semantic planes according to different interior surfaces of buildings.

[0034] It should be noted that different interior surfaces refer to walls, ceilings, and floors. In other words, the first camera image and the second camera image are divided into several semantic planes based on the walls, ceilings, and floors, respectively.

[0035] In other words, the ResNet101 network is improved by replacing the last average pooling with max pooling, and replacing the 1×1 convolutional layer with a fully connected layer for pixel classification. This allows for upsampling of the feature map through three deconvolutional layers, ultimately resulting in a semantic plane map of the same size as the original image, where different colors represent the ceiling, floor, and walls. For the network structure, please refer to [link to network diagram]. Figure 2 .

[0036] S103, extract feature points and feature descriptors from the image to be processed to obtain each feature point in the image to be processed and its corresponding feature descriptor.

[0037] As an example, feature points and feature descriptors are extracted from the image to be processed to obtain each feature point in the image to be processed and its corresponding feature descriptor. This includes: using the feature point decoder in the trained SuperPoint model to detect the location of feature points, and using sub-pixel convolution to reduce the computational load in order to extract feature points; and using the feature descriptor decoder in the trained SuperPoint model to generate the feature descriptor corresponding to each feature point in the image to be processed.

[0038] In other words, such as Figure 3 As shown, a pre-trained SuperPoint model is used to extract feature points and feature descriptors. The SuperPoint model includes an Encoder and two Decoders. The Encoder uses a VGG structure for feature encoding, and the Decoder is divided into an Interest Point Decoder and a Descriptor Decoder. The Interest Point Decoder is used for feature point location detection, and the Descriptor Decoder is used to generate feature descriptors. The Interest Point Decoder calculates the probability that each pixel in the image is a feature point, and sub-pixel convolution is used to reduce the computational cost. Semi-dense feature descriptors are first learned, then bicubic interpolation is performed to obtain the complete descriptor, and finally L2 regularization is applied to the feature descriptors to obtain the final descriptor.

[0039] S104, construct a feature point matching model, and use the semantic plane map corresponding to the first camera image, each feature point and its corresponding feature descriptor, and the semantic plane map corresponding to the second camera image, each feature point and its corresponding feature descriptor as a dataset to train the feature point matching model.

[0040] As an example, such as Figure 4As shown, training the feature point matching model includes: using positional encoding to map the positional information of each feature point into a high-dimensional vector so as to fuse the positional information with its corresponding feature descriptor; improving the receptive field of the fused feature descriptor through an attention mechanism to enhance feature communication between the first camera image and the second camera image; constructing a similarity matrix of the feature descriptors to obtain the corresponding point set of the first camera image and the second camera image; obtaining the confidence of the semantic plane map corresponding to the first camera image and the second camera image based on the corresponding point set; constructing a first loss function based on the true value of each pair of corresponding points, and constructing a second loss function based on the corresponding point set of the first camera image and the second camera image and the confidence of the semantic plane map, so as to obtain the final loss function based on the first loss function and the second loss function.

[0041] As an example, the similarity matrix of the feature descriptors is constructed according to the following formula, which is the cost matrix for feature point matching:

[0042]

[0043] Where i represents a feature point in the first camera image, j represents a feature point in the second camera image, I represents the feature point set in the first camera image, J represents the feature point set in the second camera image, and -f i f represents the eigenvector of feature point i. j Let represent the eigenvector of feature point j, and “<>” indicates the inner product; by continuously scaling and updating the SC matrix through the Sinkhorn Algorithm, the cost matrix P for matching feature points of the two images is obtained.

[0044] As an example, due to the lack of true semantic labels for the building frame, the following method is used to measure the confidence of the extracted building frame:

[0045] First, the building frames of the first and second camera images are extracted and input into the feature point matching model. The correspondence between the building frames of the two images is determined based on the obtained matching pairs.

[0046] For the corresponding point set M output by the matching network pre = {(i′, j′)|}, the planar set L of the first camera images A ={L l1 L l2 L l3 L l4 L l5} and the planar set L of the second camera images B ={L r1 L r2 L r3 L r4L r5 If i′ lies in plane L lm (L lm ∈L A In ), j′ lies in plane L rn (L rn ∈L B If a point in plane L is considered to support a plane L, then the corresponding point is considered to support the plane L. lm With L rn The viewpoint that they are on the same plane in three-dimensional space; from this, the correspondence between the first camera image plane and the second camera image plane can be obtained; if there are K pairs of corresponding points supporting L lm With L rn If they are on the same plane, then L lm With L rn The support S of the same plane (lm，rn) =K; where i′ represents a feature point in the first camera image, j′ represents a feature point in the second camera image, I represents the feature point set of the first camera image, J represents the feature point set of the second camera image, M represents the total number of feature points in the first camera image, N represents the total number of feature points in the second camera image, and (i′, j′) represents a correct pair of corresponding feature points i′ in the first camera image and j′ in the second camera image, i′∈I, j′∈J; M pre = (i′, j′) represents the corresponding points predicted by the network in a pair of images.

[0047] The support matrix S is a 5×5 matrix. The correspondence between the building frames of two images is represented by the function F, where F(S) represents the index of the row with respect to the maximum value of matrix S, and L represents the support matrix. lm With L rn They are on the same plane; thus, the correspondence between the first camera image plane and the second camera image plane is obtained. However, since this correspondence may be incorrect, the confidence level of the plane correspondence needs to be evaluated. The calculation formula is as follows:

[0048]

[0049] Where LC represents the confidence level of the plane matching, NT represents the number of corresponding points that satisfy the plane matching relationship, and NA represents the total number of corresponding points.

[0050] As an example, for the truth value M of a pair of corresponding points gt ={(i, j)}, construct the loss function

[0051]

[0052] Where i represents the i-th feature point in the first camera image, j represents the j-th feature point in the second camera image, I represents the feature point set of the first camera image, J represents the feature point set of the second camera image, M represents the total number of feature points in the first camera image, N represents the total number of feature points in the second camera image, and (i, j) indicates that the i-th feature point in the first camera image and the j-th feature point in the second camera image form a correct pair of corresponding points, i∈I, j∈J. gt = (i, j) represents all correct corresponding points in a pair of images.

[0053] For the matching M obtained by network prediction pre = {(i′, j′)} to construct the loss function

[0054]

[0055] Among them, L j L represents the plane to which the i-th feature point of the first camera image belongs. j LC represents the plane to which the j-th feature point of the second camera image belongs; LC represents the confidence level of the building frame extracted from the image; if i and j belong to the same plane, it means that the corresponding point predicted by the network may be correct, and no penalty is imposed; if i and j belong to the same plane, it means that the corresponding point predicted by the network may be wrong, and a penalty is imposed.

[0056] The final loss function formula is as follows:

[0057] Loss = Loss1 + Loss2

[0058] The entire network is optimized using the backpropagation algorithm, making the network more focused on feature points on static objects.

[0059] S105, based on the trained feature point matching model, match the camera image to be matched to obtain the corresponding points of the camera image to be matched, so as to perform camera pose estimation based on the corresponding points of the camera image to be matched.

[0060] As an example, after obtaining the corresponding points of the camera images to be matched, the data with abnormal matches is filtered out by the RANSAC algorithm.

[0061] It should be noted that the Random Sampling Consensus (RANSAC) algorithm is a commonly used method for filtering out abnormal matches. To a certain extent, this method can effectively filter out some abnormal matches, thereby solving for a more accurate camera pose.

[0062] As an example, camera pose estimation is performed based on the corresponding points of the camera images to be matched, including: using the OpenCV library to solve for the essential matrix of the corresponding points of the camera images to be matched after filtering out abnormal matching data, so as to estimate the rotation and translation of the camera images to be matched.

[0063] It should be noted that the experimental results of each method are compared in the table below:

[0064] AUC5° AUC10° AUC20° Precision MNN 6.07％ 9.17％ 13.71％ 25.12％ Superglue 21.83％ 30.11％ 36.85％ 82.67％ Ours 27.35％ 36.27％ 44.21％ 85.84％

[0065] In summary, the visual localization method based on indoor building frame constraints according to embodiments of the present invention first acquires an image to be processed, which includes a first camera image and a second camera image; then, the building frame is extracted from the image to be processed to obtain a semantic plane map of the same size as the image to be processed; simultaneously, feature points and feature descriptors are extracted from the image to be processed to obtain each feature point in the image to be processed and its corresponding feature descriptor; next, a feature point matching model is constructed, and the semantic plane map corresponding to the first camera image, each feature point and its corresponding feature descriptor, and the semantic plane map corresponding to the second camera image, each feature point and its corresponding feature descriptor are used as a dataset to train the feature point matching model; finally, the trained feature point matching model is used to match the camera images to be matched to obtain the corresponding points of the camera images to be matched, so as to perform camera pose estimation based on the corresponding points of the camera images to be matched; thus, the invariance of the building frame can be utilized to guide the network to focus on feature points on static objects, reduce the interference of dynamic objects, and thereby improve the localization effect.

[0066] In another specific embodiment, based on the visual positioning method based on indoor building frame constraints provided in the above specific embodiments, this specific embodiment provides a visual positioning system based on indoor building frame constraints.

[0067] For the positioning system provided in this specific embodiment, please refer to [link / reference]. Figure 5 It includes an acquisition module 10, a first extraction module 20, a second extraction module 30, a training module 40, and an estimation and localization module 50.

[0068] The acquisition module 10 is used to acquire the image to be processed, which includes a first camera image and a second camera image; the first extraction module 20 is used to extract the building frame of the image to be processed to obtain a semantic plane map of the same size as the image to be processed; the second extraction module 30 is used to extract feature points and feature descriptors of the image to be processed to obtain each feature point in the image to be processed and its corresponding feature descriptor; the training module 40 is used to construct a feature point matching model and train the feature point matching model using the semantic plane map corresponding to the first camera image, each feature point and its corresponding feature descriptor, and the semantic plane map corresponding to the second camera image, each feature point and its corresponding feature descriptor as a dataset; the estimation and localization module 50 is used to match the camera images to be matched based on the trained feature point matching model to obtain the corresponding points of the camera images to be matched, so as to perform camera pose estimation based on the corresponding points of the camera images to be matched.

[0069] As an example, the first extraction module 20 is also used to extract architectural frames from the image to be processed based on the improved ResNet101 network, so as to segment the image to be processed into a semantic plane map composed of multiple small semantic planes according to different interior surfaces of buildings.

[0070] As an example, the second extraction module 30 is further configured to: use the feature point decoder in the trained SuperPoint model to detect the location of feature points, and use sub-pixel convolution to reduce the computational load in order to extract feature points; and use the feature descriptor decoder in the trained SuperPoint model to generate a feature descriptor corresponding to each feature point in the image to be processed.

[0071] As an example, the training module 40 is further configured to: map the positional information of each feature point into a high-dimensional vector using positional encoding, so as to fuse the positional information with its corresponding feature descriptor; enhance the receptive field of the fused feature descriptor through an attention mechanism, so as to enhance the feature communication between the first camera image and the second camera image; construct a similarity matrix of feature descriptors to obtain the corresponding point set of the first camera image and the second camera image; obtain the confidence of the semantic plane map corresponding to the first camera image and the second camera image based on the corresponding point set; construct a first loss function based on the true value of each pair of corresponding points, and construct a second loss function based on the corresponding point set of the first camera image and the second camera image and the confidence of the semantic plane map, so as to obtain the final loss function based on the first loss function and the second loss function.

[0072] It should be noted that the above description and examples of the visual positioning method based on indoor building frame constraints also apply to the visual positioning system based on indoor building frame constraints in this embodiment, and will not be repeated here.

[0073] The visual positioning system based on indoor building frame constraints provided by the present invention can utilize the invariance of the building frame to guide the network to focus on feature points on static objects, reduce interference from dynamic objects, and thus improve the positioning effect.

[0074] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0075] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0077] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0078] It should be noted that any reference signs placed between parentheses in the claims should not be construed as limiting the claims. The word "comprising" does not exclude the presence of components or steps not listed in the claims. The word "a" or "an" preceding a component does not exclude the presence of a plurality of such components. The invention can be implemented by means of hardware comprising several different components and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0079] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0080] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

[0081] In the description of this invention, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.

[0082] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0083] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0084] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms should not be construed as necessarily referring to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0085] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A visual positioning method based on indoor building frame constraints, characterized in that, Includes the following steps: Acquire an image to be processed, wherein the image to be processed includes a first camera image and a second camera image; The building frame is extracted from the image to be processed to obtain a semantic planar graph with the same size as the image to be processed; Feature points and feature descriptors are extracted from the image to be processed to obtain each feature point in the image to be processed and its corresponding feature descriptor; A feature point matching model is constructed, and the semantic plane graph corresponding to the first camera image, each feature point and its corresponding feature descriptor, and the semantic plane graph corresponding to the second camera image, each feature point and its corresponding feature descriptor are used as datasets to train the feature point matching model; The trained feature point matching model is used to match the camera image to be matched in order to obtain the corresponding points of the camera image to be matched, so as to perform camera pose estimation based on the corresponding points of the camera image to be matched. The process includes extracting the building framework from the image to be processed to obtain a semantic planar graph of the same size as the image to be processed, including: The image to be processed is extracted based on the improved ResNet101 network, so that the image to be processed is segmented into a semantic plane map composed of multiple small semantic planes according to different interior surfaces of buildings. Specifically, the last average-pooling in ResNet is replaced with max-pooling, and the 1×1 convolutional layer is replaced with a fully connected layer to classify pixels. The training of the feature point matching model includes: Position encoding is used to map the position information of each feature point into a high-dimensional vector so that the position information can be fused with its corresponding feature descriptor; The receptive field of the fused feature descriptor is enhanced by using an attention mechanism to improve feature exchange between the first and second camera images. Construct a similarity matrix of feature descriptors to obtain the corresponding point set of the first camera image and the second camera image; The confidence levels of the semantic plane graphs corresponding to the first camera image and the second camera image are obtained based on the corresponding point set; A first loss function is constructed based on the true values ​​of each pair of corresponding points, and a second loss function is constructed based on the set of corresponding points of the first camera image and the second camera image and the confidence of the semantic plane graph, so as to obtain the final loss function based on the first loss function and the second loss function.

2. The visual positioning method based on indoor building frame constraints as described in claim 1, characterized in that, Feature point and feature descriptor extraction is performed on the image to be processed to obtain each feature point in the image to be processed and its corresponding feature descriptor, including: Feature point location detection is performed using the feature point decoder in the trained SuperPoint model, and sub-pixel convolution is used to reduce the computational load in order to extract feature points. The feature descriptor decoder in the trained SuperPoint model is used to generate the feature descriptor corresponding to each feature point in the image to be processed.

3. The visual positioning method based on indoor building frame constraints as described in claim 1, characterized in that, After obtaining the corresponding points of the camera images to be matched, the data with abnormal matches are filtered out by the RANSAC algorithm.

4. The visual positioning method based on indoor building frame constraints as described in claim 3, characterized in that, Camera pose estimation is performed based on the corresponding points of the camera image to be matched, including: For the corresponding points of the camera image to be matched after filtering out abnormal matching data, the essential matrix is ​​solved using the OpenCV library in order to estimate the rotation and translation of the camera image to be matched.

5. A visual positioning system based on indoor building frame constraints, characterized in that, include: An acquisition module is used to acquire an image to be processed, wherein the image to be processed includes a first camera image and a second camera image; The first extraction module is used to extract the building frame from the image to be processed in order to obtain a semantic planar map with the same size as the image to be processed. The second extraction module is used to extract feature points and feature descriptors from the image to be processed, so as to obtain each feature point in the image to be processed and its corresponding feature descriptor. The training module is used to construct a feature point matching model and train the feature point matching model using the semantic plane map corresponding to the first camera image, each feature point and its corresponding feature descriptor, and the semantic plane map corresponding to the second camera image, each feature point and its corresponding feature descriptor as a dataset. The localization estimation module is used to match the camera image to be matched based on the trained feature point matching model to obtain the corresponding points of the camera image to be matched, so as to perform camera pose estimation based on the corresponding points of the camera image to be matched. The first extraction module is further used for, The image to be processed is extracted based on the improved ResNet101 network, so that the image to be processed is segmented into a semantic plane map composed of multiple small semantic planes according to different interior surfaces of buildings. Specifically, the last average-pooling in ResNet is replaced with max-pooling, and the 1×1 convolutional layer is replaced with a fully connected layer to classify pixels. The training module is also used for, Position encoding is used to map the position information of each feature point into a high-dimensional vector so that the position information can be fused with its corresponding feature descriptor; The receptive field of the fused feature descriptor is enhanced by using an attention mechanism to improve feature exchange between the first and second camera images. Construct a similarity matrix of feature descriptors to obtain the corresponding point set of the first camera image and the second camera image; The confidence levels of the semantic plane graphs corresponding to the first camera image and the second camera image are obtained based on the corresponding point set; A first loss function is constructed based on the true values ​​of each pair of corresponding points, and a second loss function is constructed based on the set of corresponding points of the first camera image and the second camera image and the confidence of the semantic plane graph, so as to obtain the final loss function based on the first loss function and the second loss function.

6. The visual positioning system based on indoor building frame constraints as described in claim 5, characterized in that, The second extraction module is also used for, Feature point location detection is performed using the feature point decoder in the trained SuperPoint model, and sub-pixel convolution is used to reduce the computational load in order to extract feature points. The feature descriptor decoder in the trained SuperPoint model is used to generate the feature descriptor corresponding to each feature point in the image to be processed.

Images