A multi-feature indoor visual positioning method and system based on ground segmentation network

Abstract

The present application relates to the field of indoor positioning technology and discloses a multi-feature indoor visual positioning method and system based on a ground segmentation network. The method comprises obtaining an initial image containing environmental information; extracting point features, line features, and surface features based on the initial image; matching the point features, line features, and surface features of the current frame with the point features, line features, and surface features in the previous key frame, calculating the matching error, and optimizing to obtain an estimated pose; judging whether the current frame is a new key frame based on the estimated pose; if the current frame is a new key frame, semantically segmenting the initial image based on a preset fully convolutional network model to obtain a ground semantic mask, and using the ground semantic mask to eliminate dynamic point features and dynamic line features located on the ground in the current frame; and performing local mapping operations. The present application has the effect of improving the accuracy and robustness of the positioning method and assisting intelligent robots in more accurate and faster positioning.

Description

Technical Field

[0001] The present application relates to the field of indoor positioning technology, and in particular to a multi-feature indoor visual positioning method and system based on a ground segmentation network. Background Art

[0002] With technological advancements, intelligent robots are gradually becoming part of our daily lives. Positioning is a fundamental yet crucial requirement for intelligent robots. Improved positioning capabilities will enable them to perform tasks with greater accuracy and stability. Currently, visual SLAM (Simultaneous Localization and Mapping) technology uses cameras to acquire environmental information, perform real-time positioning, and build maps. Compared to satellite positioning and lidar, visual SLAM offers the advantages of low cost and ease of deployment. It can also operate in complex environments, breaking away from the limitations of satellite positioning, which is limited to outdoor environments. Depending on the front-end, visual SLAM can be primarily categorized as feature point SLAM and direct SLAM. Feature point SLAM first uses feature point detection and descriptor extraction to identify significant feature points in the image. These feature points are then used for camera pose estimation and map construction. The extracted feature points are invariant and can be matched across frames. Common feature points include corners and edges. However, due to limitations in feature point extraction and matching, feature point SLAM performs poorly in areas with low texture, repetitive texture, or missing texture. Direct SLAM uses image pixel values ​​for motion estimation and map construction. It estimates camera motion by minimizing the brightness error between images and uses the gradient of image brightness to estimate the depth of the map, eliminating the need for pre-detection and feature extraction. However, direct SLAM faces the challenge of its strong assumption of constant grayscale, making it sensitive to strong exposure, drastic lighting changes, and dynamic objects.

[0003] In indoor environments, when faced with low-texture or texture-missing conditions such as large expanses of white walls, the point features extracted by feature point SLAM are not stable enough and are easily affected by environmental changes. This situation results in the SLAM system lacking sufficiently stable and reliable information when achieving positioning, which in turn leads to a decrease in positioning accuracy and easily causes tracking failures. Furthermore, the numerous light sources in indoor environments will produce many "false" projections in areas prone to specular reflections, such as tiles. The so-called "false" projections mean that the positions of these projections in the actual three-dimensional space are not fixed, but will change with the relative motion of the light source and the camera. Feature points and feature lines are then extracted from these projections for subsequent pose estimation. Since the positions of these feature points and feature lines change dynamically, using them for pose estimation will reduce positioning accuracy.

[0004] Regarding the above-mentioned related technologies, the inventors found that the existing visual SLAM technology has the problem of low positioning accuracy due to lack of reliable information in indoor environment scenarios with low texture, repeated texture, missing texture, and easy specular reflection. Summary of the Invention

[0005] In order to improve the positioning accuracy in indoor environments with low texture, repeated texture, missing texture, and easy specular reflection, the present application provides a multi-feature indoor visual positioning method and system based on a ground segmentation network.

[0006] In a first aspect, the present application provides a multi-feature indoor visual positioning method based on a ground segmentation network.

[0007] This application is achieved through the following technical solutions:

[0008] A multi-feature indoor visual positioning method based on ground segmentation network, comprising the following steps:

[0009] Obtain an initial image containing environmental information;

[0010] Extracting point features, line features, and surface features based on the initial image;

[0011] Match the point features, line features, and surface features of the current frame with the point features, line features, and surface features of the previous key frame, calculate the matching error, and obtain the estimated pose through nonlinear optimization;

[0012] According to the estimated pose, determining whether the current frame is a new key frame;

[0013] If the current frame is a new keyframe, semantic segmentation is performed on the initial image based on a preset fully convolutional network model to obtain a ground semantic mask, and the ground semantic mask is used to remove dynamic point features and dynamic line features located on the ground in the current frame to obtain an optimized frame;

[0014] A local mapping operation is performed using the optimized frame.

[0015] In a preferred example, the present application may be further configured as follows: based on the initial image, the step of extracting point features includes:

[0016] Dividing the initial image into layers according to a pyramid to obtain a target grid of a fixed size located on the initial image;

[0017] Setting a FAST threshold and extracting a preset number of FAST corner points for each target grid;

[0018] If the number of FAST corner points extracted from any target grid is less than a preset number, lowering the FAST threshold of the target grid until the number of FAST corner points extracted from the target grid is equal to the preset number;

[0019] Point features are determined from the FAST corner points.

[0020] In a preferred example, the present application may be further configured as follows: the step of determining point features from the FAST corner points includes:

[0021] Determining whether the width of the initial image exceeds a width threshold;

[0022] If the width of the initial image exceeds the width threshold, the initial image is cropped into a target size area, wherein:

[0023] Determining the number of the FAST corner points in the node;

[0024] If the number of the FAST corner points is greater than 1, the node is split into four child nodes based on the quadtree method, and the child nodes without FAST corner points are removed;

[0025] Determine the number of the FAST corner points in the child node;

[0026] If the number of FAST corner points is greater than 1, the sub-nodes are further split using a quadtree method, and sub-nodes without FAST corner points are removed;

[0027] Until the child node can no longer be split, or the total number of child nodes exceeds the preset node threshold;

[0028] Select a FAST corner point that meets the requirements from each child node and summarize it to obtain the point features.

[0029] In a preferred example, the present application may be further configured as follows: based on the initial image, the step of extracting line features includes:

[0030] Using an enhanced edge rendering algorithm, the initial image is subjected to noise suppression through Gaussian filtering to obtain a de-noised image;

[0031] Calculating the gradient magnitude and edge direction map of the denoised image;

[0032] extracting anchor points of the denoised image based on the gradient magnitude and the edge direction map;

[0033] The anchor points are connected using an enhanced routing algorithm to obtain line features of the initial image, wherein if the edge direction from the previous pixel point to the current pixel point changes, the pixel point that generates the diagonal line is deleted from the candidate edge points.

[0034] In a preferred example, the present application may be further configured as follows: before the step of connecting the anchor points using an enhanced routing algorithm to obtain the line features of the initial image, the method further includes:

[0035] Test candidate line segments of different lengths to obtain the line segment lengths of the candidate line segments;

[0036] Preset a jump length, and determine whether the length of any candidate line segment is greater than the jump length;

[0037] When the line segment length of the candidate line segment is greater than the jump length, determining whether another pixel aligned with the candidate line segment and having a target number of pixels to be jumped from the current pixel is located in the initial image, wherein the target number of pixels to be jumped equals the length of the jump length;

[0038] When another pixel aligned with the candidate line segment and having a target number of jump pixels from the current pixel is located in the initial image, determining whether there are at least a target number of extended pixels when connecting the anchor points along the edge direction, wherein the target number of extended pixels equals the length of the jump length;

[0039] When there are at least a target number of extended pixels when connecting the anchor points along the edge direction, determining whether the extended pixels are completely aligned with the candidate line segment;

[0040] When the extended pixels are completely aligned with the candidate line segments, a preset minimum discontinuity portion is automatically skipped when connecting anchor points to obtain the line features of the initial image.

[0041] In a preferred example, the present application may be further configured as follows: based on the initial image, the step of extracting surface features includes:

[0042] According to the depth map of the initial image, a two-dimensional pixel back-projection operation is performed on the target pixel points, and then converted into a three-dimensional point set to obtain a structured point cloud sequence;

[0043] Dividing the depth map of the initial image into blocks of a preset size to obtain divided blocks, wherein each divided block corresponds to a node in the graph model;

[0044] Based on the structure point cloud sequence, a plane extraction algorithm based on principal component analysis is used to calculate the three-dimensional center coordinates and covariance matrix of the points in each node;

[0045] Solving the minimum eigenvalue of the covariance matrix and taking the minimum eigenvalue as the mean square error of the node;

[0046] Eliminate nodes whose mean square error is greater than a preset value, nodes with depth missing pixels, nodes with depth discontinuous edges, and nodes located at the boundary between two planes;

[0047] Sort the nodes in descending order according to the mean square error of the nodes, and establish a minimum heap structure;

[0048] For the node with the minimum mean square error in the minimum heap structure, traverse the neighborhood of the node, find the neighboring node with the minimum mean square error after merging, perform a merging operation, and remove the node and the neighboring nodes from the minimum heap structure;

[0049] Determine whether the mean square error of the merged nodes is less than the preset value;

[0050] If the mean square error of the merged node is less than or equal to the preset value, adding the merged node to the minimum heap structure;

[0051] If the mean square error of the merged node is greater than the preset value, the merged node is eliminated;

[0052] Until all nodes are extracted using the plane extraction algorithm based on principal component analysis, the minimum heap structure is used as the surface feature.

[0053] In a preferred example, the present application can be further configured as follows: the step of determining whether the mean square error of the merged nodes is less than the preset value includes:

[0054] Get the center coordinates of the node with the minimum mean square error in the minimum heap structure The center coordinates of the neighborhood node with the smallest mean square error after merging in,

[0055] Calculate the center coordinates of the node with the minimum mean square error in the minimum heap structure and the node after merging the neighboring nodes in, k1 is the number of three-dimensional points of the node with the minimum mean square error in the minimum heap structure, and k2 is the number of three-dimensional points of the neighborhood nodes;

[0056] Calculate the covariance matrix Σ3 of the node with the minimum mean square error in the minimum heap structure and the node after merging the neighboring nodes, where Σ1 is the covariance matrix of the node with the minimum mean square error in the minimum heap structure, and Σ2 is the covariance matrix of the neighborhood nodes;

[0057] Calculate the mean square error of the merged nodes based on the center coordinates and covariance matrix of the merged nodes;

[0058] Determine whether the mean square error of the merged nodes is less than the preset value.

[0059] In a preferred example, the present application can be further configured as follows:

[0060] Get the current scene image in real time;

[0061] The ORB feature extraction algorithm is used to describe the attributes of the feature points of the current scene image to obtain the ORB feature point descriptor;

[0062] The K-means++ algorithm is used to cluster the ORB feature point descriptors and construct a K-tree structure dictionary;

[0063] Searching for a word in the structure dictionary according to the ORB feature point descriptor, and calculating TF-IDF as the weight of the word to form a bag-of-words vector of the current scene image;

[0064] Calculate the similarity between the bag-of-words vector of the current scene image and the bag-of-words vector of the saved keyframe;

[0065] When the similarity exceeds a preset loop threshold and satisfies the loop determination strategy, it is determined to be a loop, triggering global optimization of all key frames and feature points.

[0066] In a second aspect, the present application provides a multi-feature indoor visual positioning system based on a ground segmentation network.

[0067] This application is achieved through the following technical solutions:

[0068] A multi-feature indoor visual positioning system based on ground segmentation network, including:

[0069] A tracking module is configured to obtain an initial image containing environmental information; extract point features, line features, and surface features based on the initial image; and match the point features, line features, and surface features of the current frame with the point features, line features, and surface features of the previous key frame, respectively, to obtain an estimated pose of the current frame.

[0070] a semantic segmentation module, configured to determine whether the current frame is a new key frame based on the estimated pose; if the current frame is a new key frame, perform semantic segmentation on the initial image to obtain a ground semantic mask, and use the ground semantic mask to remove dynamic point features and dynamic line features located on the ground in the current frame to obtain an optimized frame;

[0071] The local mapping module is used to perform a local mapping operation using the optimized frame.

[0072] In a preferred example, the present application can be further configured as follows:

[0073] A loop detection module is configured to acquire a current scene image in real time; describe the attributes of feature points in the current scene image to obtain feature point descriptors; cluster the feature point descriptors to construct a structured dictionary; search for words in the structured dictionary based on the feature point descriptors and calculate word weights to form a bag-of-words vector for the current scene image; calculate the similarity between the bag-of-words vector of the current scene image and the bag-of-words vector of a saved key frame; and determine a loop when the similarity exceeds a preset loop threshold and satisfies a loop determination strategy.

[0074] The global BA module is used to trigger global optimization of all keyframes and feature points when a loop is detected.

[0075] In summary, compared with the prior art, the technical solution provided by this application has at least the following beneficial effects:

[0076] Acquire an initial image containing environmental information to perceive the external environment; based on the initial image, extract point features, line features and surface features, match the point features, line features and surface features of the current frame with the point features, line features and surface features in the previous key frame respectively, calculate the matching error, and obtain the estimated pose through nonlinear optimization. Tightly couple point features, line features and surface features for pose estimation to solve the problem of insufficient feature points affecting positioning accuracy in indoor texture-deficient scenes and improve the robustness of the positioning method; according to the estimated pose, determine whether the current frame is a new key frame. If the current frame is a new key frame, perform semantic segmentation on the initial image based on the preset full convolutional network model to obtain a ground semantic mask, and use the ground The semantic mask eliminates the dynamic point features and dynamic line features located on the ground in the current frame to eliminate the dynamic point features and dynamic line features in the ground area with severe reflection. It responds to the influence of positioning accuracy caused by a large number of dynamic projections in areas prone to mirror reflections such as tiles due to the large number of indoor light sources, and ensures that the point features and line features used for pose estimation are statically stable, thereby improving the accuracy of the positioning method. Based on the fully convolutional structure, the fully convolutional network model can process input images of any size, can be trained directly from pixel to pixel, realize pixel-level semantic understanding, and can process images very quickly; finally, the optimized frame is used to perform local mapping operations to assist intelligent robots in more accurate and faster positioning. BRIEF DESCRIPTION OF THE DRAWINGS

[0077] Figure 1 A schematic diagram of the overall process of a multi-feature indoor visual positioning method based on a ground segmentation network is provided as an exemplary embodiment of the present application.

[0078] Figure 2A schematic diagram of the principle of determining point features using a quadtree method in a multi-feature indoor visual positioning method based on a ground segmentation network is provided as another exemplary embodiment of the present application.

[0079] Figure 3 A schematic diagram of the principle of connecting anchor points using an intelligent routing algorithm in a multi-feature indoor visual positioning method based on a ground segmentation network provided in another exemplary embodiment of the present application.

[0080] Figure 4 A schematic structural diagram of a fully convolutional network model based on a multi-feature indoor visual positioning method based on a ground segmentation network provided as an exemplary embodiment of the present application.

[0081] Figure 5 A schematic diagram of eliminating dynamic point features and dynamic line features located on the ground in a multi-feature indoor visual positioning method based on a ground segmentation network provided as an exemplary embodiment of the present application.

[0082] Figure 6 A structural block diagram of a multi-feature indoor visual positioning system based on a ground segmentation network is provided as an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0083] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, those skilled in the art may make non-creative modifications to the present embodiment as needed, but as long as they are within the scope of the claims of the present application, they are protected by the patent law.

[0084] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0085] In this document, the term "and / or" simply describes a relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document, unless otherwise specified, generally indicates an "or" relationship between the related objects.

[0086] It has been found that there are two prominent problems in using visual SLAM positioning indoors.

[0087] The first problem is that point-based visual SLAM systems often fail due to their lack of robustness in low-texture artificial environments. Therefore, it is very important to provide more robust and reliable feature information and use this information together for visual SLAM positioning to improve the robustness of the system.

[0088] The second problem is that most indoor environments are paved with ceramic tiles that are prone to specular reflections, and the numerous indoor light sources often result in a large amount of light and shadows on the ground. For example, there are many projections of light tubes and ventilation ducts on the ground. The positions of these projections in the actual three-dimensional space are not fixed, but will change with the relative movement of the light source and the camera, forming dynamic light and shadows. In indoor environments with dynamic light and shadows, the visual SLAM system easily extracts dynamic point features and dynamic line features composed of dynamic light and shadows, and performs pose estimation based on the matching results of these dynamic features, resulting in a significant reduction in positioning accuracy. Therefore, how to eliminate this dynamic feature information is very important for indoor visual SLAM systems with a static environment assumption. Eliminating this dynamic feature information can significantly improve indoor visual positioning accuracy.

[0089] The embodiments of the present application are described in further detail below with reference to the accompanying drawings.

[0090] Reference Figure 1 , an embodiment of the present application provides a multi-feature indoor visual positioning method based on a ground segmentation network, and the main steps of the method are described as follows.

[0091] S1: Obtain an initial image containing environmental information;

[0092] S2: Based on the initial image, extract point features, line features and surface features;

[0093] S3: Match the point features, line features, and surface features of the current frame with the point features, line features, and surface features of the previous key frame, calculate the matching error, and obtain the estimated pose through nonlinear optimization;

[0094] S4: judging whether the current frame is a new key frame according to the estimated pose;

[0095] S5: If the current frame is a new key frame, semantic segmentation is performed on the initial image based on a preset fully convolutional network model to obtain a ground semantic mask map, and the ground semantic mask map is used to remove dynamic point features and dynamic line features located on the ground in the current frame to obtain an optimized frame;

[0096] S6: Perform a local mapping operation using the optimized frame.

[0097] With the rapid development and cost reduction of sensor technologies, technologies such as the Global Navigation Satellite System (GNSS), cameras, inertial navigation, LiDAR, and LiDAR millimeter-wave radar are now widely used in intelligent robotics. These sensors enable intelligent robots to obtain richer and more in-depth information from their environment, enabling better operations and decision-making. With the improvement of computing power and the expansion of storage capacity, various intelligent robots can now process large amounts of sensor data in real time, greatly enhancing the functionality and computing power of intelligent robots and enabling them to more effectively analyze and understand their surroundings.

[0098] Using visual SLAM (Simultaneous Localization and Mapping) technology, an intelligent robot's RGB-D camera captures image information of its surroundings, obtaining an initial image containing environmental information to perceive the external environment, locate itself in real time, and construct a map. This intelligent robot can be an unmanned vehicle, drone, or other similar vehicle. Compared to satellite positioning and LiDAR technologies, visual SLAM technology offers advantages in low cost, ease of deployment, and ability to operate in complex environments, breaking free from the limitations of satellite positioning, which is limited to outdoor environments.

[0099] The basic functions of visual SLAM technology include real-time positioning and map construction, which uses image data collected by the camera to perceive the environment, locate its own position and create a unit map at the same time.

[0100] Specifically, point features are extracted based on the initial image.

[0101] Point features are obtained by using OpenCV's ORB feature extraction, but this method will cause the feature points to be too dense and concentrated, resulting in a clustering of feature points, and reduce the amount of information during closed-loop detection, thereby reducing the accuracy of SLAM.

[0102] To improve the density and concentration of point feature distribution and make it uniform, the initial image is divided into several small grids, and the best n manually selected feature points are retained in each grid. However, this method has the following problems: if a certain grid cannot extract enough feature points (such as textureless areas), the number of feature points extracted from the entire image cannot meet the extraction requirements, which can seriously lead to SLAM failure.

[0103] In order to extract a sufficient number of point features, in one embodiment, based on the initial image, the step of extracting point features includes:

[0104] Dividing the initial image into layers according to a pyramid to obtain a target grid of a fixed size located on the initial image;

[0105] Setting a FAST threshold and extracting a preset number of FAST corner points for each target grid;

[0106] If the number of FAST corner points extracted from any target grid is less than a preset number, lowering the FAST threshold of the target grid until the number of FAST corner points extracted from the target grid is equal to the preset number;

[0107] Point features are determined from the FAST corner points.

[0108] Specifically, the initial image is processed by pyramid layering, and for each layer, the initial image is divided into target grids of a fixed size, which may be 30×30 pixels.

[0109] Set a FAST threshold and extract a preset number of FAST corner points for each target grid. A FAST corner point is defined as one that is located in a different region from a sufficient number of pixels in its surrounding area. For example, if a pixel is located in a different region from all eight pixels in its surrounding area, it is considered a FAST corner point.

[0110] When the preset number of FAST corner points cannot be extracted for any target grid, the FAST threshold is automatically lowered until the preset number of FAST corner points can be extracted for the target grid, ensuring that the preset number of FAST corner points can be extracted even in areas with weak or no texture, thereby ensuring the number of point features and avoiding the problem of SLAM failure caused by the number of point features extracted from the entire image not meeting the requirements.

[0111] In one embodiment, the step of determining point features from the FAST corner points includes:

[0112] Determining whether the width of the initial image exceeds a width threshold;

[0113] If the width of the initial image exceeds the width threshold, the initial image is cropped into a target size area, wherein:

[0114] Determining the number of the FAST corner points in the node;

[0115] If the number of the FAST corner points is greater than 1, the node is split into four child nodes based on the quadtree method, and the child nodes without FAST corner points are removed;

[0116] Determine the number of the FAST corner points in the child node;

[0117] If the number of FAST corner points is greater than 1, the sub-nodes are further split using a quadtree method, and sub-nodes without FAST corner points are removed;

[0118] Until the child node can no longer be split, or the total number of child nodes exceeds the preset node threshold;

[0119] Select a FAST corner point that meets the requirements from each child node and summarize it to obtain the point features.

[0120] A quadtree-based approach is used to uniformly select FAST corner points to determine point features. If the image is wide, the image is first divided into left and right w / h nodes. For example, a 640×480 pixel image has only one node.

[0121] If the number of feature points in a node is greater than 1, each node is divided into four sub-nodes. If there is no feature point in a sub-node, the sub-node is deleted.

[0122] Continue to split the child nodes until they cannot be split any further or the total number of child nodes exceeds the node threshold.

[0123] Finally, a FAST corner point with the best quality is manually selected from each split child node and summarized to obtain the point features.

[0124] Reference Figure 2 The principle of determining point features from FAST corner points using a quadtree-based method is shown in the figure. The node threshold N=25 is set to retain 25 child nodes.

[0125] First, the initial image is divided into w / h nodes. For example, a 640×480 pixel image has only one node.

[0126] The first split is performed, and 1 node node is split into 4 child nodes node<25. At this time, further splitting is required.

[0127] The second split is performed, and the 4 child nodes node are split into 15 child nodes node<25. Because one of the child nodes has no FAST corner points and is removed, further splitting is required.

[0128] The third split is performed to remove the child nodes without FAST corner points. The 15 child nodes node are split into 30 child nodes node>25. At this time, the total number of child nodes is greater than the preset node threshold 25, and the split is ended.

[0129] Finally, the best quality FAST corner point is selected from each child node and summarized to obtain the point feature.

[0130] Based on the initial image, line features are extracted.

[0131] By adopting the edge drawing algorithm (ED), which is suitable for grayscale images, the line features of the initial image are extracted using the following steps: 1) suppressing the noise of the initial image through Gaussian filtering; 2) calculating the gradient magnitude of the processed initial image and obtaining the edge direction map; 3) extracting anchor points based on the gradient magnitude and edge direction map; 4) connecting the anchor points through an intelligent routing algorithm to obtain the edge map of the image.

[0132] Specifically, a 5×5 Gaussian kernel convolution with σ=1 is used to suppress the noise of the initial image and obtain the denoised image. For the gradient magnitude and direction calculation, the Sobel operator is first applied to calculate the horizontal gradient G of the denoised image. x and vertical gradient G y , and then use the L1 norm, that is, G=|G x |+|G y |. Set the gradient threshold and set G=0 for pixels below the gradient threshold. Quantize the gradient direction O into two values: vertical edge and horizontal edge. x |≥|G y | is a vertical edge, and when |G x |<|G y | is a horizontal edge. The anchor point is the pixel where the line segment is drawn. To extract the anchor point pixel, scan the image pixels where G>0 and calculate whether the gradient magnitude G is a local maximum along the gradient quantization direction O. When the pixel direction O(x,y) corresponds to a vertical edge and satisfies G[x,y]-G[x-1,y]≥T anchor and G[x,y]-G[x+1,y]≥T anchor When , the pixel is the anchor point. The horizontal edge calculation method in the vertical direction is similar to this, and will not be repeated here. In this embodiment, it is also possible to increase T anchor The values ​​and spacing of columns and rows of scanned pixels limit the number of anchor points.

[0133] Reference Figure 3 , connecting anchor points through intelligent routing algorithms. The working principle of intelligent routing process is as follows:

[0134] Start at the anchor point and look at the direction of the edge that passes through the anchor point. If the horizontal edge passes through the anchor point, connect it by going left and right, such as Figure 3 (b) If the vertical edge passes through the anchor point, it is connected upwards and then downwards, as shown in Figure 3 (c) During the movement, only three direct neighbor anchors are considered and the one with the largest gradient value is selected.

[0135] The process stops in two cases:

[0136] (1) Move out of the edge area, that is, the threshold gradient value of the current pixel is 0;

[0137] (2) Encountering a previously detected edge point. A simple example is Figure 3 As shown in (a), the numbers in the pixels are the gradient magnitudes; black pixels represent threshold pixels, anchor points are marked with red circles, and pixels collected during the connection process are marked with yellow circles. Assume that the pixel at (8,4) is the starting anchor point. Since the horizontal edge passes through this anchor point, a horizontal walk is initiated (first to the left, then to the right). The small arrows indicate the pixels considered during the wiring process, and the blue arrows indicate the selected pixels. Figure 3 (b) and Figure 3 As shown in (c), at each iteration, the three immediate neighboring anchor points in the edge direction are examined, and the pixel with the largest gradient value is selected. At (3,5), the edge direction changes from horizontal to vertical. At this point, the current horizontal leftward walk is stopped, and two vertical walks are initiated: one upward and one downward. The vertical upward walk terminates immediately because it will select the edge segment pixel (4,4) that has already been selected. The vertical downward walk continues until (5,10). In this way, a portion of the image edge is obtained starting from a single anchor point. By performing this operation on each anchor point, all the image edges can be obtained.

[0138] The ED algorithm itself can extract high-quality edges of an image, but to obtain line features in the image, additional processing of the edge map of the image is required, performing line segment detection on the edge map of the image to obtain line features.

[0139] In order to increase the detection speed of line features and improve the efficiency of line segment detection, in one embodiment, based on the initial image, the step of extracting line features mainly includes an enhanced edge drawing (EED) algorithm. The specific steps are as follows:

[0140] Performing noise suppression on the initial image by Gaussian filtering to obtain a noise-reduced image;

[0141] Calculating the gradient magnitude and edge direction map of the denoised image;

[0142] extracting anchor points of the denoised image based on the gradient magnitude and the edge direction map;

[0143] The anchor points are connected using an enhanced routing algorithm to obtain line features of the initial image, wherein if the edge direction from the previous pixel to the current pixel changes, the pixel points that generate diagonal lines, that is, the pixel points that will cause the edge to be non-straight, are deleted from the candidate edge points.

[0144] The EED algorithm is based on the ED algorithm. The intelligent routing algorithm in ED is improved into an enhanced routing algorithm. The other steps remain unchanged, so as to combine the edge drawing and line segment detection processes in one step, which greatly improves the efficiency of line feature extraction.

[0145] To improve the intelligent routing algorithm, the EED algorithm uses an enhanced routing algorithm for line segment extraction. During the drawing process, the EED algorithm considers the previous pixel and the current pixel. During the movement process, as long as the edge direction from the previous pixel to the current pixel does not change, that is, there is no change from vertical to horizontal or from horizontal to vertical, the same pixels as in the ED algorithm will be explored. If the edge direction changes, the existing ED algorithm will draw an edge that breaks the straight line assumption, that is, the pixel point goes back to form a corner, generating some diagonal pixels. However, the EED algorithm uses a different method to ensure that the edge points are always along a line. The specific method mainly removes pixels that will produce diagonal lines from the candidate edge points.

[0146] By using the EED algorithm to extract line features, it has two advantages: 1) it is faster than the original ED routing algorithm because fewer pixels need to be explored, and 2) it avoids the meaningless situation of searching for line segments.

[0147] When a line segment is interrupted by multiple discontinuous points, the length of the discontinuity is unknown, which will affect the efficiency of line feature extraction.

[0148] In order to skip these parts when discontinuities are detected and continue drawing along the line segment direction as much as possible, thereby improving the efficiency of line feature extraction, in one embodiment, before the step of connecting the anchor points using the enhanced routing algorithm to obtain the line features of the initial image, the method further includes:

[0149] Test candidate line segments of different lengths to obtain the line segment lengths of the candidate line segments;

[0150] Preset a jump length, and determine whether the length of any candidate line segment is greater than the jump length;

[0151] When the line segment length of the candidate line segment is greater than the jump length, determining whether another pixel aligned with the candidate line segment and having a target number of pixels to be jumped from the current pixel is located in the initial image, wherein the target number of pixels to be jumped equals the length of the jump length;

[0152] When another pixel aligned with the candidate line segment and having a target number of jump pixels from the current pixel is located in the initial image, determining whether there are at least a target number of extended pixels when connecting the anchor points along the edge direction, wherein the target number of extended pixels equals the length of the jump length;

[0153] When there are at least a target number of extended pixels when connecting the anchor points along the edge direction, determining whether the extended pixels are completely aligned with the candidate line segment;

[0154] When the extended pixels are completely aligned with the candidate line segments, a preset minimum discontinuity portion is automatically skipped when connecting anchor points to obtain the line features of the initial image.

[0155] In this embodiment, since a 5×5 Gaussian smoothing kernel is used for noise suppression, any pixel discontinuity will affect a neighborhood of at least 5 pixels. Therefore, the minimum discontinuity length threshold is set to 5 pixels. The jump length J∈[5,7,9] is preset, that is, the number of pixels is preset to 5, 7, or 9.

[0156] Test candidate line segments of different lengths, determine the length between any two anchor points, and check whether the line segment length of the candidate line segment is greater than the jump length. If all of the following conditions are true, the anchor point drawing process will continue after detecting the discontinuity:

[0157] 1) The length of the line segment is greater than the number of pixels to jump to, J; 2) The pixel a aligned with the line segment and J pixels away from the current pixel c is located inside the image, and the gradient G[a]>0; 3) Starting from pixel a, EED can draw at least J pixels along the edge direction, and these J pixels are called extended pixels; 4) The extended pixels are completely aligned with the line segment, that is, the starting pixel and the ending pixel of the J extended pixels coincide with the starting and ending points of the preset line segment, and the remaining extended pixels are located on the preset line segment.

[0158] Based on the initial image, face features are extracted.

[0159] In one embodiment, the step of extracting face features based on the initial image includes:

[0160] According to the depth map of the initial image, a two-dimensional pixel back-projection operation is performed on the target pixel points, and then converted into a three-dimensional point set to obtain a structured point cloud sequence;

[0161] Dividing the depth map of the initial image into blocks of a preset size to obtain divided blocks, wherein each divided block corresponds to a node in the graph model;

[0162] Based on the structure point cloud sequence, a plane extraction algorithm based on principal component analysis is used to calculate the three-dimensional center coordinates and covariance matrix of the points in each node;

[0163] Solving the minimum eigenvalue of the covariance matrix and taking the minimum eigenvalue as the mean square error of the node;

[0164] Eliminate nodes whose mean square error is greater than a preset value, nodes with depth missing pixels, nodes with depth discontinuous edges, and nodes located at the boundary between two planes;

[0165] Sort the nodes in descending order according to the mean square error of the nodes, and establish a minimum heap structure;

[0166] For the node with the minimum mean square error in the minimum heap structure, traverse the neighborhood of the node, find the neighboring node with the minimum mean square error after merging, perform a merging operation, and remove the node and the neighboring nodes from the minimum heap structure;

[0167] Determine whether the mean square error of the merged nodes is less than the preset value;

[0168] If the mean square error of the merged node is less than or equal to the preset value, adding the merged node to the minimum heap structure;

[0169] If the mean square error of the merged node is greater than the preset value, the merged node is eliminated;

[0170] Until all nodes are extracted using the plane extraction algorithm based on principal component analysis, the minimum heap structure is used as the surface feature.

[0171] In one embodiment, the step of determining whether the mean square error of the merged nodes is less than the preset value includes:

[0172] Get the center coordinates P1 of the node with the minimum mean square error in the minimum heap structure and the center coordinates P2 of the neighboring node with the minimum mean square error after merging, where P1 = [x1, y1, z1] T , P2=[x2,y2,z2] T ;

[0173] Calculate the center coordinates of the node with the minimum mean square error in the minimum heap structure and the node after merging the neighboring nodes in, k1 is the number of three-dimensional points of the node with the minimum mean square error in the minimum heap structure, and k2 is the number of three-dimensional points of the neighborhood nodes;

[0174] Calculate the covariance matrix Σ3 of the node with the minimum mean square error in the minimum heap structure and the node after merging the neighboring nodes, where Σ1 is the covariance matrix of the node with the minimum mean square error in the minimum heap structure, and Σ2 is the covariance matrix of the neighborhood nodes;

[0175] Calculate the mean square error of the merged nodes based on the center coordinates and covariance matrix of the merged nodes; solve the minimum eigenvalue of the covariance matrix and use it as the mean square error of the merged nodes;

[0176] Determine whether the mean square error of the merged nodes is less than the preset value.

[0177] Specifically, the PEAC plane extraction algorithm is used to extract the surface features of the initial image.

[0178] The PEAC plane extraction algorithm is based on principal component analysis and hierarchical clustering. This algorithm leverages the structure of the point cloud and uses hierarchical clustering and principal component analysis to cluster the point cloud into blocks to extract planes. The specific process can be divided into the following steps:

[0179] 1) Constructing a structural point cloud: By inputting a depth map, perform two-dimensional pixel back-projection on the pixels with valid depth information, and then convert them into a three-dimensional point set according to the pinhole camera model to form an ordered sequence of structural point clouds. Valid depth means that the depth is not 0 or negative. The pinhole camera model is a simplified camera model. Its basic assumption is that light is projected onto the image plane through a small hole, and the camera's intrinsic parameters (intrinsic parameters) and extrinsic parameters (extrinsic parameters) are used to describe the camera's position, posture, and imaging process. Two pictures of the same scene are taken by two cameras, and the parallax between corresponding points in the image is calculated. The depth information is calculated using the parallax, and finally converted into a three-dimensional point set.

[0180] 2) Graph initialization: The depth map is divided into 16×16 blocks, each of which corresponds to a node in the graph model. For each node, a plane is fitted using principal component analysis to calculate the 3D center coordinates of the points within the node. and the covariance matrix Σ. Next, the minimum eigenvalue of the covariance matrix is ​​calculated and used as the mean square error of the node. Finally, nodes with a mean square error greater than a preset value are eliminated, as well as nodes with missing depth pixels, nodes with discontinuous depth edges (if there is a discontinuous depth edge in a node, that is, the change in depth value is too drastic, which can be determined by detecting the difference in the depth values ​​of adjacent pixels, then the node is eliminated) and nodes located at the boundary of two planes (which can be determined by fitting the plane difference between the node and the surrounding nodes. If the difference is large, the node is eliminated), so that the graph model is initialized.

[0181] 3) Hierarchical clustering: For the nodes in the graph model, sort them in descending order according to the size of the mean square error, and establish a minimum heap structure to quickly find the node with the minimum mean square error. According to the principle of principal component analysis, the node with the minimum mean square error is more likely to represent a plane. Therefore, the merging operation is started from the node with the minimum mean square error. For the node with the minimum mean square error in the minimum heap, traverse its neighborhood and find the neighboring node with the smallest mean square error after merging. Then, perform a merge operation to obtain the merged node, and remove the two nodes from the minimum heap structure. If the mean square error of the merged node is less than the preset value, the merged node is added to the minimum heap structure; otherwise, the node is removed from the graph model. Repeat the above steps until all nodes are extracted and the graph model construction process is completed.

[0182] In one embodiment, after the step of dividing the depth map of the initial image into blocks of a preset size to obtain divided blocks, the method further includes:

[0183] For the over-segmented plane, the graph model is reconstructed using the segmented blocks around it to obtain new nodes around the over-segmented plane;

[0184] A hierarchical clustering algorithm is used to cluster and merge the new nodes to obtain optimized nodes of the over-segmented plane.

[0185] For over-segmented planes, which occur when a large plane is cut into multiple small blocks, a plane is a collection of multiple continuous 16*16 pixel blocks. These small blocks are independently fitted into the plane, which may lead to increased errors and inaccurate models. In this case, the surrounding segments (i.e., the surrounding segmented pixel blocks) can be reconstructed into nodes, and then the graph model can be rebuilt. Then, a hierarchical clustering algorithm can be used to cluster and merge the surrounding segments (nodes) to obtain a larger plane, which can better improve the over-segmented plane situation and obtain more accurate surface feature extraction results.

[0186] Also, at the edge, in order to reduce the jagged phenomenon of the surface features, the erosion algorithm can be used to process the nodes at the edge, such as merging some of the surrounding nodes to eliminate unnecessary fine structures.

[0187] Also, for unused (culled) nodes, you can consider merging them into nearby planes. This helps optimize the structure of the graph model and better utilizes the information in the graph data.

[0188] Next, the extracted point features, line features, and surface features of the current frame are matched with the point features, line features, and surface features of the previous key frame, respectively, the matching error is calculated, and the pose estimation is obtained through nonlinear optimization. The nonlinear optimization algorithm can use the Loevenberg-Marquardt algorithm, conjugate gradient method, gradient descent method, etc.

[0189] Based on the pose estimation results, determine whether the current frame is a new keyframe. The view change value is obtained by calculating the relative pose change between the current frame and the most recent keyframe. If the view change value exceeds the preset keyframe threshold, the current frame is considered a new keyframe.

[0190] If the current frame is determined to be a key frame, a new thread is started to perform semantic segmentation on the image using the preset fully convolutional network model FCN to obtain a ground semantic mask map.

[0191] In one embodiment, the steps of constructing the fully convolutional network model include:

[0192] Convert the input initial image into a higher-level feature representation through an encoder;

[0193] progressively decoding the feature representation of the encoder into a prediction result of the same size as the initial image through a decoder;

[0194] The output of the target layer of the encoder is connected to the input of the corresponding layer of the decoder through a jump, and the encoder uses features from different levels, adopts a softmax function for prediction, and outputs a ground semantic mask map.

[0195] Reference Figure 4 The overall grid architecture based on the fully convolutional network model (FCN) is an encoder-decoder structure for image segmentation tasks. Input represents the input image, Conv represents the convolutional layer, Deconv represents the deconvolutional layer, and Plane params represents the plane parameters, which represent the loss of the difference from the label value to assist in the training of the semantic segmentation network.

[0196] The encoder is responsible for converting the input image into a higher-level feature representation. Higher-level features are those with higher dimensions than the input image. If the features encoded by the encoder are sufficient to represent the parameters of a plane, the encoder is considered to have strong representational capabilities.

[0197] The decoder is responsible for gradually decoding the converted feature representation into a prediction result of the same size as the original input image.

[0198] The outputs of some layers of the encoder are directly connected to the corresponding layers of the decoder to implement skip connection Concat, so that more low-level feature information can be integrated during the decoding process, thereby improving the accuracy of segmentation.

[0199] In the decoder, features from different levels are used simultaneously for multi-scale prediction, so that the network can process feature information of different scales at the same time, improve the recognition ability of objects of different sizes, and help the network better capture the details and global information in the image, thereby improving the accuracy and robustness of segmentation.

[0200] During training, the hyperparameters can be set to β1 = 0.99, β2 = 0.9999, epoch = 4, and lr = 0.0001. The model is trained using the training dataset provided by SYNTHIA.

[0201] Due to its fully convolutional structure, FCN can process input images of any size, can be trained directly from pixel to pixel, achieve pixel-level semantic understanding, and can process images very quickly.

[0202] In one embodiment, the predicted image can also be scaled by using a random factor in the range of [1, 1.15], such as an upsampling operation Upsample to enlarge the spatial resolution (i.e., width and height) of the predicted image to make the image resolution higher, and then randomly cropping to enhance the image data.

[0203] The softmax function is used during prediction to convert the network output into a probability distribution, which facilitates the classification of each pixel. This design combines the powerful features and advantages of convolutional neural networks and provides an efficient and effective solution for image segmentation tasks, allowing us to more accurately extract ground semantic masks.

[0204] In one embodiment, except for the prediction layer, the fully convolutional network model uses the ReLU activation function for learning. The use of the ReLU activation function throughout the network helps learn nonlinear features, thereby improving the representation ability of the model.

[0205] Reference Figure 5 Furthermore, the ground semantic mask is used to eliminate the dynamic point features and dynamic line features located on the ground in the current frame. The ground semantic information of the ground semantic mask is used to eliminate the dynamic point features and dynamic line features existing in the ground area with severe reflection, so as to ensure that the point features and line features used for pose estimation are statically stable, thereby improving the accuracy of system positioning.

[0206] In one embodiment, the step of removing dynamic point features located on the ground in the current frame using the ground semantic mask includes:

[0207] Determine whether the pixel corresponding to the point feature in the key frame falls within the range of the ground semantic pixel in the ground semantic mask;

[0208] If the pixel corresponding to the point feature in the key frame falls within the range of the ground semantic pixels in the ground semantic mask, it is determined to be a dynamic point feature located on the ground in the key frame and is removed.

[0209] For point features, the relationship between their pixel coordinates and the pixel values ​​of the ground segmentation semantic map is used to determine whether they need to be removed. If the pixel corresponding to the point feature falls within the range of the ground semantic pixels in the ground segmentation semantic map, the feature point is considered unstable and needs to be removed, such as Figure 5 (a) is shown; otherwise, it is retained. The gray part in the figure is the ground semantics, the red points are the point features located on the ground, and the green points are not located on the ground.

[0210] In one embodiment, the step of removing the dynamic line features located on the ground in the current frame using the ground semantic mask includes:

[0211] Uniformly sampling the line features in the key frame to obtain a number of equally spaced sampling points;

[0212] Determine whether the pixel corresponding to the sampling point falls within the range of ground semantic pixels in the ground semantic mask;

[0213] Counting the number of sampling points that fall within the range of ground semantic pixels in the ground semantic mask;

[0214] If the number of sampling points falling within the range of the ground semantic pixels in the ground semantic mask exceeds a preset rejection threshold, they are determined to be dynamic line features located on the ground in the key frame and are rejected.

[0215] For line features, we first sample them uniformly. In this example, we get 11 equally spaced sampling points. Then, for each sampling point, we use the same method as for point features to determine whether the sampling point falls on the ground. Finally, we count the number of sampling points that fall on the ground. If the total number of sampling points is greater than a set threshold, such as 6, then the line feature is considered to be on the ground, unstable, and needs to be removed. Figure 5 (b) shows that otherwise it is retained. The gray part in the figure represents the ground semantics, the red points are the point features located on the ground, and the green points are not located on the ground.

[0216] Finally, the current frame without the dynamic point features and dynamic line features on the ground, that is, the optimized frame, is used to perform local mapping operations, including using the initial frame and its feature points as the initial state of the local map; adding new feature points to the local map when they are detected to become new map points, and matching feature points between adjacent frames; and using the observation perspective of the optimized frame to optimize the position of the map points to obtain the local map.

[0217] In one embodiment, the local map is optimized to further improve the accuracy of pose estimation. This optimization can utilize a local bundle adjustment (BA) operation, which helps correct and optimize the relative pose relationship between the current keyframe and its surrounding frames to ensure the consistency and accuracy of the entire map. Optimizing the local map with bundle adjustment can better correct errors, resulting in more reliable results during subsequent positioning and navigation.

[0218] In order to maximize the real-time performance and efficiency of positioning, in one embodiment, a multi-feature indoor visual positioning method based on a ground segmentation network further includes the following steps:

[0219] Get the current scene image in real time;

[0220] The ORB feature extraction algorithm is used to describe the attributes of the feature points of the current scene image to obtain the ORB feature point descriptor;

[0221] The K-means++ algorithm is used to cluster the ORB feature point descriptors and construct a K-tree structure dictionary;

[0222] Searching for a word in the structure dictionary according to the ORB feature point descriptor, and calculating TF-IDF as the weight of the word to form a bag-of-words vector of the current scene image;

[0223] Calculate the similarity between the bag-of-words vector of the current scene image and the bag-of-words vector of the saved keyframe;

[0224] When the similarity exceeds a preset loop threshold and satisfies the loop determination strategy, it is determined to be a loop, triggering global optimization of all key frames and feature points.

[0225] Specifically, the intelligent robot collects the current scene image, clusters the ORB feature point descriptors of the current scene image using the K-means++ algorithm, and constructs a K-tree structure dictionary.

[0226] Then, based on the ORB feature point descriptors of the current image, the word is searched in the dictionary and the TF-IDF is calculated as the weight of each word to form the bag-of-words vector for the current image. TF-IDF (term frequency–inverse document frequency) is a common weighting technique used in information retrieval and data mining. TF stands for term frequency, and IDF stands for inverse document frequency.

[0227] Next, the similarity between the bag-of-words vector of the current scene image and the bag-of-words vectors of all saved keyframes is calculated using methods such as the L1 norm. If the similarity score is sufficiently high and meets the preset loop detection strategy, for example, the similarity is above a set threshold and meets the spatial relationship conditions, it is determined to be a loop, meaning that the intelligent robot has revisited a scene similar to one it has previously experienced. Loop detection can help intelligent robots better identify and utilize previous experience during navigation, improving the accuracy and robustness of positioning and path planning, and maximizing the real-time and efficiency of positioning.

[0228] When a loop is detected, a global bundle adjustment (BA) optimization is triggered for all keyframes and feature points. Global BA involves optimizing all keyframes and feature points in the entire constructed map to maximize the overall pose estimation result. This global BA optimization allows the system to more accurately correct the relative pose relationships between keyframes, thereby improving the accuracy and stability of the entire system.

[0229] In summary, a multi-feature indoor visual positioning method based on a ground segmentation network obtains an initial image containing environmental information to perceive the external environment; based on the initial image, point features, line features and surface features are extracted, and the point features, line features and surface features of the current frame are matched with the point features, line features and surface features in the previous key frame respectively, and the matching error is calculated. The estimated pose is obtained through nonlinear optimization, and the point features, line features and surface features are tightly coupled for pose estimation to solve the problem of insufficient feature points affecting positioning accuracy in indoor texture-deficient scenes, thereby improving the robustness of the positioning method; according to the estimated pose, it is determined whether the current frame is a new key frame. If the current frame is a new key frame, the preset full The convolutional network model performs semantic segmentation on the initial image to obtain a ground semantic mask map, and uses the ground semantic mask map to eliminate the dynamic point features and dynamic line features located on the ground in the current frame, so as to eliminate the dynamic point features and dynamic line features existing in the ground area with severe reflections. This is to cope with the impact of a large number of dynamic projections caused by the large number of indoor light sources and areas such as tiles that are prone to mirror reflections, and to ensure that the point features and line features used for pose estimation are statically stable, thereby improving the accuracy of the positioning method. The full convolutional network model is a lightweight ground semantic segmentation neural network with faster detection speed. Finally, the optimized frame is used to perform local mapping operations to assist the intelligent robot in more accurate and faster positioning.

[0230] A multi-feature indoor visual localization method based on ground segmentation network proposes a method of tightly coupling multiple features. Since line features and surface features are more stable than point features, three different types of features are tightly coupled. Point features (orb), line features (elsed) and surface features (PEAC) are combined for pose estimation and environment mapping to solve the problem that insufficient number of point features in indoor environments easily leads to visual SLAM failure and improve the robustness of the system.

[0231] To address the problem that a large number of dynamic projections appear in ground areas with numerous indoor light sources and prone to specular reflections, resulting in the extracted point and line features not conforming to the static assumption, a multi-feature indoor visual positioning method based on a ground segmentation network constructs a lightweight ground semantic segmentation neural network to extract ground semantic information from images. The ground semantic information is used to eliminate dynamic point features and dynamic line features in ground areas with severe reflections. The FCN architecture network is used for ground semantic segmentation, and only ground point and line features located in key frames are eliminated. This not only ensures that the features used for pose estimation are statically stable, thereby improving positioning accuracy; but also ensures the real-time performance of the system by only performing semantic segmentation operations on key frames.

[0232] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0233] Reference Figure 6 A multi-feature indoor visual positioning method based on a ground segmentation network is implemented by a multi-feature indoor visual positioning system based on a ground segmentation network. The overall framework of a multi-feature indoor visual positioning system based on a ground segmentation network is similar to a SLAM system using feature points, including:

[0234] The tracking module is used to extract point features, line features, and surface features from the input initial image containing environmental information; match the point features, line features, and surface features of the current frame with the point features, line features, and surface features of the previous key frame respectively to obtain the estimated pose of the current frame;

[0235] a semantic segmentation module, configured to determine whether the current frame is a new key frame based on the estimated pose; if the current frame is a new key frame, perform semantic segmentation on the initial image to obtain a ground semantic mask, and use the ground semantic mask to remove dynamic point features and dynamic line features located on the ground in the current frame to obtain an optimized frame;

[0236] The local mapping module is used to perform a local mapping operation using the optimized frame.

[0237] By adding line features (Enhanced Line Segment Drawing, ELSED) and surface features (FastPlane Extraction Using Agglomerative Hierarchical Clustering, PEAC), the system tightly couples point, line, and surface features for pose estimation, addressing the problem of insufficient feature points in texture-poor indoor scenes. Specifically, during the tracking phase, point, line, and surface features are extracted for each image frame. These features are then matched with the corresponding features from the previous keyframe to produce a pose estimate for the current frame.

[0238] At the same time, in order to cope with the impact of a large number of dynamic projections caused by the large number of indoor light sources and areas prone to mirror reflections such as tiles, this system uses a semantic segmentation module to extract the ground semantic mask map from the key frame, and removes the dynamic point features and dynamic line features located in the ground semantic mask map to obtain an optimized frame to improve the accuracy of pose estimation.

[0239] According to the result of pose estimation, it can be determined whether the current optimized frame is a new key frame. If the current optimized frame is a new key frame, local mapping is performed, new map points and map lines are created, and local BA is performed to optimize the local pose estimation results.

[0240] Detect and remove redundant keyframes.

[0241] A multi-feature indoor visual positioning system based on a ground segmentation network, further comprising:

[0242] A loop detection module is configured to acquire a current scene image in real time; describe the attributes of feature points in the current scene image to obtain feature point descriptors; cluster the feature point descriptors to construct a structured dictionary; search for words in the structured dictionary based on the feature point descriptors and calculate word weights to form a bag-of-words vector for the current scene image; calculate the similarity between the bag-of-words vector of the current scene image and the bag-of-words vector of a saved key frame; and determine a loop when the similarity exceeds a preset loop threshold and satisfies a loop determination strategy, thereby maximizing the real-time performance and efficiency of the system.

[0243] The global BA module is used to trigger global optimization of all keyframes and feature points when a loop is detected.

[0244] The loop detection module is executed in parallel with other modules as a daemon process. This parallel execution mode enables the system to achieve a good balance between real-time performance and accuracy, thereby better responding to positioning and navigation requirements in different scenarios and complex environments.

[0245] The loop detection module also includes loop correction function.

[0246] The embodiment of the present application also provides a multi-feature indoor visual positioning device based on a ground segmentation network, which corresponds one-to-one to the multi-feature indoor visual positioning method based on a ground segmentation network in the above embodiment. The multi-feature indoor visual positioning device based on a ground segmentation network includes:

[0247] An acquisition module, used to obtain an initial image containing environmental information;

[0248] A feature extraction module, configured to extract point features, line features, and surface features based on the initial image;

[0249] The fusion module is used to match the point features, line features, and surface features of the current frame with the point features, line features, and surface features of the previous key frame, calculate the matching error, and obtain the estimated pose through nonlinear optimization;

[0250] A key frame detection module, configured to determine whether a current frame is a new key frame based on the estimated pose;

[0251] an optimization module configured to, if the current frame is a new keyframe, perform semantic segmentation on the initial image based on a preset fully convolutional network model to obtain a ground semantic mask, and use the ground semantic mask to remove dynamic point features and dynamic line features located on the ground in the current frame to obtain an optimized frame;

[0252] A mapping module is used to perform a local mapping operation using the optimized frame.

[0253] A multi-feature indoor visual positioning device based on a ground segmentation network also includes:

[0254] The loop detection module is used to obtain the current scene image in real time; use the ORB feature extraction algorithm to describe the attributes of the feature points of the current scene image to obtain ORB feature point descriptors; use the K-means++ algorithm to cluster the ORB feature point descriptors to construct a K-tree structure dictionary; search for words in the structure dictionary based on the ORB feature point descriptors, and calculate TF-IDF as the weight of the words to form a bag-of-words vector of the current scene image; calculate the similarity between the bag-of-words vector of the current scene image and the bag-of-words vector of the saved key frame; when the similarity exceeds a preset loop threshold and meets the loop determination strategy, it is determined to be a loop, triggering global optimization of all key frames and feature points.

[0255] A multi-feature indoor visual positioning device based on a ground segmentation network also includes:

[0256] The model construction module is used to convert the input initial image into a higher-level feature representation through an encoder; gradually decode the feature representation of the encoder into a prediction result of the same size as the initial image through a decoder; and connect the output of the target layer of the encoder to the input of the corresponding layer of the decoder through a jump. The encoder uses features from different levels and adopts a softmax function for prediction to output a ground semantic mask.

[0257] For the specific limitations of a multi-feature indoor visual positioning device based on a ground segmentation network, please refer to the above limitations of a multi-feature indoor visual positioning method based on a ground segmentation network, which will not be repeated here.

[0258] Each module in the multi-feature indoor visual positioning device based on a ground segmentation network can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a computer device's memory in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0259] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, a memory, a network interface, and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, any of the above-mentioned multi-feature indoor visual positioning methods based on a ground segmentation network is implemented.

[0260] In one embodiment, a computer-readable storage medium is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, any one of the multi-feature indoor visual positioning methods based on a ground segmentation network is implemented.

[0261] In one embodiment, a computer program product is provided, which includes a computer program, and when the computer program is executed by a processor, it implements any of the above-mentioned multi-feature indoor visual positioning methods based on a ground segmentation network.

[0262] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium, including a number of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. When the computer program is executed, it may include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0263] Those skilled in the art will clearly understand that for the sake of convenience and brevity in description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the system can be divided into different functional units or modules to complete all or part of the functions described above.

Claims

1. A multi-feature indoor visual positioning method based on ground segmentation network, characterized in that: The following steps are included: Obtain an initial image containing environmental information; Based on the initial image, point features, line features and surface features are extracted; wherein, based on the initial image, the step of extracting line features includes: Using an enhanced edge rendering algorithm, the initial image is subjected to noise suppression through Gaussian filtering to obtain a de-noised image; Calculating the gradient magnitude and edge direction map of the denoised image; extracting anchor points of the denoised image based on the gradient magnitude and the edge direction map; Connecting the anchor points using an enhanced routing algorithm to obtain line features of the initial image, wherein if the edge direction from the previous pixel to the current pixel changes, the pixel point that generates the diagonal line is deleted from the candidate edge points; Match the point features, line features, and surface features of the current frame with the point features, line features, and surface features of the previous key frame, calculate the matching error, and obtain the estimated pose through nonlinear optimization; According to the estimated pose, determining whether the current frame is a new key frame; If the current frame is a new keyframe, semantic segmentation is performed on the initial image based on a preset fully convolutional network model to obtain a ground semantic mask, and the ground semantic mask is used to remove dynamic point features and dynamic line features located on the ground in the current frame to obtain an optimized frame; performing a local mapping operation using the optimized frame; Before the step of connecting the anchor points using the enhanced routing algorithm to obtain the line features of the initial image, the method further includes: Test candidate line segments of different lengths to obtain the line segment lengths of the candidate line segments; Preset a jump length, and determine whether the length of any candidate line segment is greater than the jump length; When the line segment length of the candidate line segment is greater than the jump length, determining whether another pixel aligned with the candidate line segment and having a target number of pixels to be jumped from the current pixel is located in the initial image, wherein the target number of pixels to be jumped equals the length of the jump length; When another pixel aligned with the candidate line segment and having a target number of jump pixels from the current pixel is located in the initial image, determining whether there are at least a target number of extended pixels when connecting the anchor points along the edge direction, wherein the target number of extended pixels equals the length of the jump length; When there are at least a target number of extended pixels when connecting the anchor points along the edge direction, determining whether the extended pixels are completely aligned with the candidate line segment; When the extended pixels are completely aligned with the candidate line segments, a preset minimum discontinuity portion is automatically skipped when connecting anchor points to obtain the line features of the initial image.

2. The multi-feature indoor visual positioning method based on ground segmentation network according to claim 1 is characterized in that Based on the initial image, the step of extracting point features includes: Dividing the initial image into layers according to a pyramid to obtain a target grid of a fixed size located on the initial image; Setting a FAST threshold and extracting a preset number of FAST corner points for each target grid; If the number of FAST corner points extracted from any target grid is less than a preset number, lowering the FAST threshold of the target grid until the number of FAST corner points extracted from the target grid is equal to the preset number; Point features are determined from the FAST corner points.

3. The multi-feature indoor visual positioning method based on ground segmentation network according to claim 2 is characterized in that The step of determining point features from the FAST corner points comprises: Determining whether the width of the initial image exceeds a width threshold; If the width of the initial image exceeds the width threshold, the initial image is cut into a target size area, wherein the area within the target size area is Determine the number of FAST corner points in the node; If the number of the FAST corner points is greater than 1, the node is split into four child nodes based on the quadtree method, and the child nodes without FAST corner points are removed; Determine the number of the FAST corner points in the child node; If the number of FAST corner points is greater than 1, the sub-nodes are further split using a quadtree method, and sub-nodes without FAST corner points are removed; Until the child node can no longer be split, or the total number of child nodes exceeds the preset node threshold; Select a FAST corner point that meets the requirements from each child node and summarize it to obtain the point features.

4. The multi-feature indoor visual positioning method based on ground segmentation network according to claim 1 is characterized in that Based on the initial image, the step of extracting face features includes: According to the depth map of the initial image, a two-dimensional pixel back-projection operation is performed on the target pixel points, and then converted into a three-dimensional point set to obtain a structured point cloud sequence; Dividing the depth map of the initial image into blocks of a preset size to obtain divided blocks, wherein each divided block corresponds to a node in the graph model; Based on the structure point cloud sequence, a plane extraction algorithm based on principal component analysis is used to calculate the three-dimensional center coordinates and covariance matrix of the points in each node; Solving the minimum eigenvalue of the covariance matrix and taking the minimum eigenvalue as the mean square error of the node; Eliminate nodes whose mean square error is greater than a preset value, nodes with depth missing pixels, nodes with depth discontinuous edges, and nodes located at the boundary between two planes; Sort the nodes in descending order according to the mean square error of the nodes, and establish a minimum heap structure; For the node with the minimum mean square error in the minimum heap structure, traverse the neighborhood of the node, find the neighboring node with the minimum mean square error after merging, perform a merging operation, and remove the node and the neighboring nodes from the minimum heap structure; Determine whether the mean square error of the merged nodes is less than the preset value; If the mean square error of the merged node is less than or equal to the preset value, adding the merged node to the minimum heap structure; If the mean square error of the merged node is greater than the preset value, the merged node is eliminated; Until all nodes are extracted using the plane extraction algorithm based on principal component analysis, the minimum heap structure is used as the surface feature.

5. The multi-feature indoor visual positioning method based on ground segmentation network according to claim 4 is characterized in that: The step of determining whether the mean square error of the merged nodes is less than the preset value includes: Get the center coordinates of the node with the minimum mean square error in the minimum heap structure The center coordinates of the neighborhood node with the smallest mean square error after merging in, Calculate the center coordinates of the node with the minimum mean square error in the minimum heap structure and the node after merging the neighboring nodes in, k1 is the number of three-dimensional points of the node with the minimum mean square error in the minimum heap structure, and k2 is the number of three-dimensional points of the neighborhood nodes; Calculate the covariance matrix ∑3 of the node with the minimum mean square error in the minimum heap structure and the node after merging the neighboring nodes, where ∑1 is the covariance matrix of the node with the minimum mean square error in the minimum heap structure, and ∑2 is the covariance matrix of the neighborhood nodes; Calculate the mean square error of the merged nodes based on the center coordinates and covariance matrix of the merged nodes; Determine whether the mean square error of the merged nodes is less than the preset value.

6. The multi-feature indoor visual positioning method based on ground segmentation network according to claim 1 is characterized in that The following steps are also included: Get the current scene image in real time; The ORB feature extraction algorithm is used to describe the attributes of feature points in the current scene image to obtain the ORB feature point descriptor; The K-means++ algorithm is used to cluster the ORB feature point descriptors and construct a K-tree structure dictionary; Searching for a word in the structure dictionary according to the ORB feature point descriptor, and calculating TF-IDF as the weight of the word to form a bag-of-words vector of the current scene image; Calculate the similarity between the bag-of-words vector of the current scene image and the bag-of-words vector of the saved keyframe; When the similarity exceeds the preset loop threshold and satisfies the loop determination strategy, it is determined to be a loop, triggering the global optimization of all key frames and feature points.

7. A multi-feature indoor visual positioning system based on ground segmentation network, characterized in that: The method for executing the multi-feature indoor visual positioning method based on ground segmentation network according to claim 1 comprises: Tracking module, used to obtain the initial image containing environmental information; Extracting point features, line features, and surface features based on the initial image; matching the point features, line features, and surface features of the current frame with the point features, line features, and surface features of the previous key frame, respectively, to obtain an estimated pose of the current frame; a semantic segmentation module, configured to determine whether the current frame is a new key frame based on the estimated pose; if the current frame is a new key frame, perform semantic segmentation on the initial image to obtain a ground semantic mask, and use the ground semantic mask to remove dynamic point features and dynamic line features located on the ground in the current frame to obtain an optimized frame; The local mapping module is used to perform a local mapping operation using the optimized frame.

8. The multi-feature indoor visual positioning system based on ground segmentation network according to claim 7 is characterized in that: Also includes, Loop detection module, used to obtain the current scene image in real time; Describing the attributes of feature points in the current scene image to obtain feature point descriptors; clustering the feature point descriptors to construct a structure dictionary; Searching for words in the structured dictionary based on the feature point descriptor and calculating the weights of the words to form a bag-of-words vector of the current scene image; calculating the similarity between the bag-of-words vector of the current scene image and the bag-of-words vector of the saved key frame; and determining a loop when the similarity exceeds a preset loop threshold and satisfies a loop determination strategy; The global BA module is used to trigger global optimization of all keyframes and feature points when a loop is detected.

Images