An indoor semantic localization method combining floor plan information

Abstract

The present invention provides an indoor semantic positioning method that combines the semantic information of a monocular image with floor plan information. The method uses OCR detection to identify business signs in a monocular image of a scene, determines the mapping relationship between two-dimensional feature sub-elements and three-dimensional map points, completes the real-world anchoring of a sparse point cloud map and an indoor floor plan, and preferably uses an ORB-SLAM algorithm to compare the image sequence to be tested with a keyframe library for repositioning, thereby achieving the determination of the three-dimensional coordinates of the mobile terminal camera position and realizing an efficient semantic SLAM function. Through the method of the present invention, it is possible to overcome the problems of high computational cost and poor scene adaptability of SLAM technology that integrates semantic information in complex scenes, and realize low-computing-power, high-robustness semantic synchronous positioning and mapping of mobile terminals that integrate indoor floor plans. The method is suitable for use by mobile devices such as mobile phones in scenes such as underground parking lots and indoor supermarkets.

Description

Technical Field

[0001] The present invention belongs to the technical field of simultaneous positioning and mapping, and particularly relates to an indoor semantic positioning method combined with floor plan information. Background Art

[0002] Indoor scenes typically feature weak external GNSS signals, three-dimensional maps, and complex environments. Furthermore, users' positioning accuracy requirements indoors are much higher than those in outdoor scenes, making it impossible to rely on traditional positioning technologies for positioning and navigation. Simultaneous Localization and Mapping (SLAM) is a technology that uses robots or other mobile devices to autonomously navigate in unknown environments. SLAM technology uses sensors (such as lidar and cameras) to acquire environmental information while integrating the robot's own motion information to achieve real-time mapping and positioning of the environment. SLAM technology has been widely used in various fields, including driverless cars, smart homes, and industrial automation.

[0003] Semantic SLAM is an advanced SLAM algorithm that integrates scene semantic information. Traditional SLAM systems rely primarily on geometric information for environment modeling and positioning, while ignoring the semantic information within the environment. In contrast, semantic SLAM leverages prior knowledge to incorporate semantic information about objects within the environment into the SLAM system. Through techniques such as object detection, recognition, and tracking, semantic information about objects in the environment is acquired and integrated with geometric information, enabling the robot to better understand the characteristics and structure of the environment, thereby improving the accuracy of environment modeling and positioning.

[0004] Semantic SLAM has been widely used in fields such as autonomous driving, smart homes, and robot navigation. The application of this technology can enhance robots' environmental perception and autonomous navigation capabilities, bringing more possibilities to future intelligent life and industrial production.

[0005] SLAM technology based on semantic segmentation offers advantages such as improved scene understanding, more accurate position estimation, improved mapping, and wider application. It has become one of the key areas of current SLAM research and development, but it also inevitably suffers from operational drawbacks. Semantic segmentation requires significant computing resources, including processing and storing large amounts of image data, performing semantic segmentation, and building three-dimensional maps. This increases the challenge of real-time performance, especially when used on mobile devices such as mobile phones. The accuracy of semantic segmentation is significantly limited due to factors such as complex scene changes. This can lead to inaccurate maps and failures in SLAM systems. Semantic segmentation requires extensive annotation work to obtain high-quality semantic information. This increases the cost of developing and maintaining SLAM systems and may require specialized knowledge.

[0006] Based on the above reasons, semantic segmentation SLAM is still a challenging problem, and exploring the integration of semantic information and simultaneous positioning and mapping has become an important trend in current research and development. Summary of the Invention

[0007] The purpose of this invention is to provide an indoor semantic positioning method that combines floor plan information with strong computing power and high robustness to achieve the determination of the three-dimensional coordinates of the mobile camera position and complete efficient semantic synchronous positioning and mapping.

[0008] The indoor semantic positioning method combined with floor plan information provided by the present invention is used to accurately locate the current position of the camera when capturing an image and display the positioning trajectory results on the floor plan. The specific steps are as follows:

[0009] Step 1: Input color images into the computer and use these images to call the OCR text recognition algorithm (HU S, HE C, ZHANG C, et al. Efficient Scene Text Recognition Model Built with PaddlePaddle Framework; proceedings of the 20217th International Conference on Big Data and Information Analytics (BigDIA), F, 2021[C]. IEEE.) to obtain the text area and text content in the image; at the same time, call the SLAM algorithm (Gao Xiang, Visual SLAM 14 Lectures: From Theory to Practice (Second Edition) [M], Beijing, Electronic Industry Press, 2019.8) to obtain a sparse point cloud map;

[0010] Step 2: Using the text area obtained in step 1, calculate the FAST corner points and image pyramid to extract the feature sub-regions in the corresponding image area. Combined with the mapping relationship between the feature sub-regions and map points, the 2D text area is matched with the 3D map points, and a sparse point cloud map containing semantic information is obtained.

[0011] Step 3: Based on the semantic information, search the text recognition results in the map database to obtain the location of the location in the indoor floor plan, and achieve anchoring between the sparse point cloud map and the indoor floor plan;

[0012] Step 4: Input the image sequence to be located, use the relocation function of the lost tracking situation in the SLAM mode, compare it with the keyframe library for relocation, and obtain the 3D coordinate trajectory of the camera in the sparse point cloud map;

[0013] Step 5: Based on the map anchoring relationship obtained in step 3, obtain the transformation matrix between the keyframe trajectory coordinate axis and the floor plan coordinate to complete the keyframe trajectory coordinate transformation; use the drawing tool to draw the positioning trajectory result curve of the repositioned image on the floor plan.

[0014] Further:

[0015] In step 2, the matching between the two-dimensional text area and the three-dimensional map point is to assign the semantic information of the corresponding image area to the three-dimensional map point by means of the mapping relationship between the two-dimensional feature sub-points and the three-dimensional map point. The specific steps include the following:

[0016] Step 2-1, determining the coincidence relationship between the two-dimensional feature sub-images and the text recognition area in each frame;

[0017] Step 2-2: If the feature point is within the text area, semantic information is assigned to the corresponding two-dimensional feature sub-point to achieve plane area matching; if not, no processing is performed;

[0018] Step 2-3: Based on the observation relationship between the map points and the key frames, the average observation direction and average observation distance are updated based on the upper and lower limits of the average observation distance determined by the number of image pyramid layers.

[0019] In steps 2-4, based on the property that the feature descriptor of a map point is defined as the median of the descriptors in all observed keyframes, this descriptor has the smallest median distance to all other descriptors. By comparing the descriptor of the map point with the descriptor of the image feature point, the mapping relationship between the 2D feature and the 3D map point is obtained, and the semantic information of the 2D feature is assigned to the 3D map point.

[0020] In step 3, the anchoring of the sparse point cloud map to the indoor floor plan includes the following sub-steps:

[0021] Step 3-1: Based on the coordinate group of map points in the existing 3D point cloud map, a coordinate transformation matrix is ​​obtained using a fitting algorithm to achieve projection mapping from the 3D sparse point cloud map to the 2D point cloud map;

[0022] Step 3-2, using the distance information between the corresponding location in the indoor floor plan in the semantic information, obtain the scale relationship between the point cloud projection mapping map and the indoor floor plan;

[0023] Step 3-3, obtaining the coordinate axis direction transformation between the two-dimensional point cloud projection mapping map and the indoor floor plan according to the specified relationship between the text recognition result location and the map coordinate axis;

[0024] In step 3-4, the origin information is used in combination with the scale factor and the coordinate axis direction information to determine the transformation relationship between the sparse point cloud map and the indoor floor plan coordinate system, and the two-dimensional point cloud projection mapping map is anchored in the actual map; the location searched by the text recognition result is matched with the coordinate group in the three-dimensional point cloud map by regional matching, and the average value of the regional matching fitting result parameters is used to reduce the error value of point-to-point matching.

[0025] In step 4, the repositioning of the input image by comparing it with the key frame library is performed through a bag of words process. The key frames containing more common visual words are scored and screened. After screening, candidate key frames are obtained. The feature points of the current frame and the candidate key frames are compared one by one. When the number of matches is greater than 15, the frame is used as the matching frame of the key frame. Finally, the pose of the matching frame is estimated to obtain the rotation and translation matrices between it and the initial frame, and the detailed coordinates of the matching frame camera in the SLAM world coordinate system are obtained.

[0026] In step 5, the transformation matrix of the key frame trajectory coordinate axis and the floor plan coordinate is obtained according to the map anchoring relationship obtained in step 3, and the key frame trajectory coordinate transformation is completed; the specific process is: according to the three-dimensional coordinates of the camera in the SLAM world coordinate system, the error is minimized so that it can be projected onto a two-dimensional plane to form two-dimensional trajectory coordinates; then the map direction vector and scale scaling obtained in step 3 are used to obtain the match between the SLAM map coordinates and the floor plan, and the coordinate values ​​are converted into actual coordinates with distance units; finally, according to the actual coordinates and the longitude and latitude of the coordinate origin, they are converted into longitude and latitude coordinates, and displayed on the floor plan using the relevant API to complete the positioning result display.

[0027] Features and advantages of the present invention

[0028] In this invention, by using OCR text recognition technology to detect and identify business signs in scene images, semantic information extraction can be achieved at a low computational cost. By leveraging the mapping relationship between two-dimensional feature sub-elements and three-dimensional map points, semantic information of the two-dimensional indoor floor plan is assigned to the three-dimensional map points, achieving real-world anchoring of the sparse point cloud map and the indoor floor plan. By optimizing the ORB-SLAM2 algorithm, the monocular image sequence obtained by the mobile terminal to be positioned is compared with the keyframe library for repositioning, which enables the determination of the three-dimensional coordinates of the mobile terminal camera position, achieving efficient semantic synchronous positioning and mapping.

[0029] The present invention can overcome the problems of high computational cost and poor scene adaptability of SLAM technology that integrates semantic information in complex scenes, and realize low-computing-power and highly robust semantic synchronous positioning and mapping on the mobile terminal that integrates indoor floor plans. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 This is an overall flow chart of the indoor semantic positioning method combined with floor plan information of the present invention.

[0031] Figure 2 This is a sparse point cloud-specific store display diagram containing semantic information of the present invention.

[0032] Figure 3 This is a sparse point cloud-specific store display diagram containing semantic information of the present invention.

[0033] Figure 4 It is a planar projection diagram of the key frame trajectory and semantic points of the present invention.

[0034] Figure 5 It is a planar projection diagram of the key frame trajectory, semantic points and all map points of the present invention.

[0035] Figure 6 This is the result of coordinate transformation and dimensionality reduction of the sparse point cloud map of the present invention.

[0036] Figure 7 It is a result diagram of the relocation trajectory of the present invention drawn on the floor plan. DETAILED DESCRIPTION

[0037] The present invention will be further described below with reference to the accompanying drawings and examples.

[0038] like Figure 1 As shown, the present invention provides an indoor semantic positioning method combined with floor plan information, which is used to accurately locate the current position of the camera when shooting an image and display the positioning trajectory results on the actual map, including the following steps:

[0039] Step 1: Input color images into the computer, use these images to call the OCR text recognition algorithm to obtain the text area and text content in the image; at the same time, call the SLAM algorithm to obtain a sparse point cloud map.

[0040] Step 2: Using the text area obtained in step 1, calculate the FAST corner points and image pyramid to extract the feature sub-regions in the corresponding image area. Combined with the mapping relationship between the feature sub-regions and map points, the 2D text area is matched with the 3D map points, and a sparse point cloud map containing semantic information is obtained.

[0041] Step 2-1: First, determine the overlap between the two-dimensional feature sub-images and the text recognition area in each frame, assign semantic information to the corresponding two-dimensional feature sub-images, and achieve plane area matching;

[0042] Step 2-2: If the feature point is within the text area, semantic information is assigned to the corresponding two-dimensional feature sub-point to achieve plane area matching; if not, no processing is performed;

[0043] Step 2-3: Based on the observation relationship between the map points and the key frames, the average observation direction and average observation distance are updated based on the upper and lower limits of the average observation distance determined by the number of image pyramid layers.

[0044] In steps 2-4, the feature descriptor of a map point is defined as the median of the descriptors in all observed keyframes, and the distance between this descriptor and the median of all other descriptors is the smallest. By comparing the descriptor of the map point with the descriptor of the image feature point, the mapping relationship between the 2D feature descriptor and the 3D map point is obtained.

[0045] Step 2-5: assign the semantic information of the two-dimensional feature sub-items to the three-dimensional map points according to the mapping relationship obtained in the previous step.

[0046] Step 3: Based on the semantic information, search the map database for text recognition results to obtain the location of the location in the indoor floor plan, and achieve anchoring between the sparse point cloud map and the indoor floor plan. This includes the following sub-steps:

[0047] Step 3-1: Based on the coordinate group of map points in the existing 3D point cloud map, a coordinate transformation matrix is ​​obtained using a fitting algorithm to achieve projection mapping from the 3D sparse point cloud map to the 2D point cloud map;

[0048] Step 3-2, using the distance information between the corresponding location in the indoor floor plan in the semantic information, obtain the scale relationship between the point cloud projection mapping map and the indoor floor plan;

[0049] Step 3-3, obtaining the coordinate axis direction transformation between the two-dimensional point cloud projection mapping map and the indoor floor plan according to the specified relationship between the text recognition result location and the map coordinate axis;

[0050] Step 3-4: Use the origin information, combined with the scale factor and coordinate axis direction information to determine the transformation relationship between the sparse point cloud map and the indoor floor plan coordinate system, and anchor the two-dimensional point cloud projection mapping map in the actual map.

[0051] Step 4: Input the image sequence to be located, use the relocation function of the lost tracking situation in the SLAM mode, compare it with the key frame library for relocation, and obtain the 3D coordinate trajectory of the camera in the sparse point cloud map.

[0052] Step 5: Based on the map anchoring relationship obtained in step 3, obtain the transformation matrix between the keyframe trajectory coordinate axis and the floor plan coordinate to complete the keyframe trajectory coordinate transformation; use the drawing tool to draw the positioning trajectory result curve of the repositioned image on the floor plan.

[0053] Example:

[0054] This article takes the monocular image captured by the Huawei P50 mobile phone as input data and uses it as an example to illustrate indoor semantic positioning combined with floor plan information.

[0055] 1. Obtain image data

[0056] The Huawei P50 is equipped with a high-definition camera capable of capturing high-definition video at various frame rates and resolutions. By converting the captured video into an image sequence, processing its image timestamps and creating a timestamp file, we can use it as input for SLAM and OCR.

[0057] 2. Sparse Mapping and OCR Recognition

[0058] With the help of the ORB-SLAM2 algorithm, the image sequence converted from the video recorded by the mobile phone is input for sparse mapping to obtain a three-dimensional sparse point cloud map. Based on the DB text detection algorithm, CRNN text recognition algorithm and Resnet50 backbone network, 1304 labeled images are used to train a text detection model and a text recognition model suitable for supermarket portal sign scenes. The image sequence is input into the OCR inference network to obtain the content of the text in the image and mark its position with a rectangular box. The output is a table file in CSV format, which includes the image name, the corresponding text content in the image and the pixel coordinates of the four corner points of the corresponding rectangular box on the image, such as Figure 2 and Figure 3 The store name recognition result is shown in the lower right corner of the image. The text content here is encoded, and different text contents are represented by the floor number and sequence number.

[0059] 3. Search text recognition results and add semantic information to the sparse point cloud

[0060] Based on the text recognition results, we use a web crawler to obtain the actual location of the place represented by the text result on the map. We first preprocessed the data obtained by OCR to remove the relevant text that is not related to the store sign; and constructed a mapping relationship. Through comparison and substring comparison, we assigned an index to each text content to facilitate subsequent embedding. We extracted the key points in the 2D text area and matched them to the 3D map points corresponding to the key points. After completing the matching between the 2D image area and the 3D map points, we adjusted and obtained a sparse point cloud map containing semantic information, such as Figure 2 and Figure 3 The part of the point cloud map corresponding to the text area is added with the encoded semantic information.

[0061] 4. Repositioning based on the new input image to be positioned

[0062] The frame that needs to be relocated is added to the key frame sequence, and then relocated in the same way as the process after the SLAM process tracking fails. First, the BOW vector of the current frame is calculated and candidate key frames are selected. After selecting all key frames that share word nodes with the current frame, the maximum number of shared word nodes of a key frame is calculated, and then the minimum threshold of shared word nodes is set to 0.8 of the maximum number, and key frames that do not meet the number are screened out; the BOW similarity scores of the remaining key frames and the current frame are calculated, and all key frames that meet the highest score greater than 0.75 are returned. The current frame and the key frame are compared one by one for matching, and the matching feature points with a successful number of less than 15 are removed (matching on the same word node). When the best distance and the suboptimal distance meet a certain relationship, the match is successful. Detect the direction, and when the match is achieved, count the number of successful matches in the three main directions. Based on the matching relationship obtained above, preferably, use PnP optimization to estimate the rotation matrix R and translation matrix T of the current frame and then process it with each key frame that meets the conditions, calculate the number of extended matching points that meet the conditions and filter the key frames. When the number of matches is met, the relocation is successful. After successful relocalization, the system enters the normal SLAM mode and can then output keyframe trajectory files like in the normal SLAM mode.

[0063] 5. Map information fusion and relocation information drawing trajectory on floor plan

[0064] Search the text content to obtain the location of the location corresponding to the text, and obtain the scale relationship and coordinate axis transformation matrix between the sparse point cloud map and the actual map based on the positional relationship between the actual locations corresponding to the map points with semantic information, such as Figure 4-Figure 6 The fitting plane and projection map shown in the figure. Based on the key frame trajectory obtained in the previous step, the fitting algorithm is used to obtain the transformation matrix that projects the three-dimensional trajectory onto the two-dimensional plane. These two matrices can be used to map the relocated key frame three-dimensional trajectory results onto the two-dimensional floor plan, as shown in the figure. Figure 7 The relocation trajectory shown.

[0065] According to the indoor semantic positioning method combined with floor plan information involved in this embodiment, due to the use of the fusion of semantic information obtained by OCR and monocular image sequences, it realizes the function of precise positioning in complex indoor environments without relying on GNSS information, which can assist users to complete precise positioning in complex indoor environments, thereby laying the foundation for indoor navigation.

[0066] In summary, this embodiment's indoor semantic localization method, combined with floor plan information, integrates visual sensor data, performs OCR processing on text in real-world scenes to extract content and pixel coordinates, and uses the sparse map obtained by the VSLAM system to attach semantic information to map points corresponding to feature points in the text area and map them to the corresponding positions of the text in the actual map, completing the anchoring of the sparse point cloud to the actual map. Furthermore, the method can also perform relocalization based on the newly input image sequence and compare it with the keyframe library. Based on the map anchoring, the relocalized three-dimensional trajectory in the sparse point cloud map is converted into a two-dimensional trajectory in the floor plan, completing the localization task in complex indoor scenes.

Claims

1. An indoor semantic positioning method combined with floor plan information, characterized in that: The specific steps are: Step 1: Input color images into the computer, use these images to call the OCR text recognition algorithm to obtain the text area and text content in the image; at the same time, call the SLAM algorithm to obtain a sparse point cloud map; Step 2: Using the text area obtained in step 1, calculate the FAST corner points and image pyramid to extract the feature sub-regions in the corresponding image area. Combined with the mapping relationship between the feature sub-regions and map points, the 2D text area is matched with the 3D map points, and a sparse point cloud map containing semantic information is obtained. Step 3: Based on the semantic information, search the text recognition results in the map database to obtain the location of the location in the indoor floor plan, and achieve anchoring between the sparse point cloud map and the indoor floor plan; Step 4: Input the image sequence to be located, use the relocation function of the lost tracking situation in the SLAM mode, compare it with the keyframe library for relocation, and obtain the 3D coordinate trajectory of the camera in the sparse point cloud map; Step 5: Based on the map anchoring relationship obtained in step 3, obtain the transformation matrix between the keyframe trajectory coordinate axis and the floor plan coordinate to complete the keyframe trajectory coordinate transformation; use the drawing tool to draw the positioning trajectory result curve of the repositioned image on the floor plan.

2. The indoor semantic positioning method combined with floor plan information according to claim 1 is characterized in that: The matching of the 2D text area and the 3D map point in step 2 is to assign the semantic information of the corresponding image area to the 3D map point by means of the mapping relationship between the 2D feature sub-elements and the 3D map point. The specific steps include the following: Step 2-1, determining the coincidence relationship between the two-dimensional feature sub-images and the text recognition area in each frame; Step 2-2: If the feature point is within the text area, semantic information is assigned to the corresponding two-dimensional feature sub-point to achieve plane area matching; otherwise, no processing is performed; Step 2-3: Based on the observation relationship between the map points and the key frames, the average observation direction and average observation distance are updated based on the upper and lower limits of the average observation distance determined by the number of image pyramid layers. In steps 2-4, based on the property that the feature descriptor of a map point is defined as the median of the descriptors in all observed keyframes, this descriptor has the smallest median distance to all other descriptors. By comparing the descriptor of the map point with the descriptor of the image feature point, the mapping relationship between the 2D feature and the 3D map point is obtained, and the semantic information of the 2D feature is assigned to the 3D map point.

3. The indoor semantic positioning method combined with floor plan information according to claim 1 is characterized in that: The anchoring of the sparse point cloud map to the indoor floor plan described in step 3 includes the following sub-steps: Step 3-1: Based on the coordinate group of map points in the existing 3D point cloud map, a coordinate transformation matrix is ​​obtained using a fitting algorithm to achieve projection mapping from the 3D sparse point cloud map to the 2D point cloud map; Step 3-2, using the distance information between the corresponding location in the indoor floor plan in the semantic information, obtain the scale relationship between the point cloud projection mapping map and the indoor floor plan; Step 3-3, obtaining the coordinate axis direction transformation between the two-dimensional point cloud projection mapping map and the indoor floor plan according to the specified relationship between the text recognition result location and the map coordinate axis; Step 3-4: Using the origin information, combined with the scale factor and coordinate axis direction information, determine the transformation relationship between the sparse point cloud map and the indoor floor plan coordinate system, and anchor the 2D point cloud projection mapping map to the actual map; The matching between the location searched by the text recognition result and the coordinate group in the three-dimensional point cloud map adopts regional matching, and the average value of the regional matching fitting result parameters is used to reduce the error value of point-to-point matching.

4. The indoor semantic positioning method combined with floor plan information according to claim 1, characterized in that: In step 4, the relocalization process of comparing the input image with the key frame library is performed through the bag of words. The key frames containing more common visual words are scored and screened. After screening, candidate key frames are obtained. The feature points of the current frame and the candidate key frames are compared one by one. When the number of feature point matches is greater than 15, it is used as the matching frame of the key frame. Finally, the pose of the matching frame is estimated to obtain the rotation and translation matrices between it and the initial frame, and based on this, the detailed coordinates of the matching frame camera in the SLAM world coordinate system are obtained.

5. The indoor semantic positioning method combined with floor plan information according to claim 1, characterized in that: As described in step 5, based on the map anchoring relationship obtained in step 3, the transformation matrix of the key frame trajectory coordinate axis and the floor plan coordinate is obtained to complete the key frame trajectory coordinate transformation; the specific process is to minimize the error estimation based on the three-dimensional coordinates of the camera in the SLAM world coordinate system so that it can be projected onto a two-dimensional plane to form two-dimensional trajectory coordinates; then, the map direction vector and scale scaling obtained in step 3 are used to obtain the match between the SLAM map coordinates and the floor plan, and the coordinate values ​​are converted into actual coordinates with distance units; finally, according to the actual coordinates and the longitude and latitude of the coordinate origin, they are converted into longitude and latitude coordinates, and displayed on the floor plan using the relevant API to complete the positioning result display.

Images