Text-based semantic visual localization method

Abstract

The application relates to the technical field of visual positioning, and discloses a semantic visual positioning method based on character identification, which comprises the following steps: a preparation step, a three-dimensional map construction step, an image retrieval step, based on the three-dimensional character map constructed in the previous step, inputting a query frame to perform image retrieval and pose estimation, after obtaining the matching frame of the current frame, the relative pose between the two frames needs to be calculated, and experiment comparison is performed; a data set proposed by TextSLAM is used in the experiment; and multiple classic and best methods in the related fields of image retrieval, position recognition and visual positioning are compared. The semantic visual positioning method based on character identification uses character identification information in a scene as a feature to perform visual positioning, so that a real bag-of-words model is realized instead of the visual "bag-of-words" model used in the foregoing work; visual positioning is performed by using semantic information based on character identification, so that the positioning accuracy and efficiency can be greatly improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual positioning, in particular to a semantic visual positioning method based on text identification. BACKGROUND

[0002] The visual positioning method inputs a query frame, finds a key frame observing the same location as the query frame from a database, and calculates the relative pose of the query frame relative to the database frame, thereby obtaining the current camera pose of the query frame. The traditional visual positioning method calculates the similarity between frames by comparing local appearance features of images, such as using common features such as FAST+BRIEF, SURF, SIFT, and their descriptors, and combining visual bag of word technology to calculate the similarity between images, thereby finding the most similar database key frame to the input query frame.

[0003] Image local appearance features are easily affected by changes in the environment, such as changes in observation angle and changes in scene lighting, which can cause significant changes in image appearance, thereby affecting visual bag of word methods that rely on image local appearance, making it difficult to achieve accurate and robust visual positioning. In view of this situation, researchers have noticed that semantic targets exist in the environment, and the semantic information they carry does not change with changes in the environment, so some research work has extracted semantic information from the environment and encoded it as features to rely on such semantic features for visual positioning, achieving good results.

[0004] There is a typical semantic target in the environment - text identification, such as store signs, road signs, office plaques, etc., which have uniform geometric structures, directly convey semantic information, are widely distributed, and have a high frequency of occurrence, making them a good visual positioning marker. Some researchers[1]use text identification extracted from two-dimensional image observations to compare the semantic information with all text observations on all frames in the database, thereby obtaining image retrieval results, and then use EKF algorithm to fuse and update the pose of the database frame with the pose result calculated by the current visual odometry. However, this method does not construct a global three-dimensional map containing text information, so the use of text identification is limited to two-dimensional image text extraction results, and can only perform a time-consuming and inefficient search process containing a large number of repeated texts. The efficiency and effectiveness of this method are far from the requirements of practical applications, and it cannot compare the similarity between images based on text identification information; therefore, we propose a semantic visual positioning method based on text identification. SUMMARY

[0005] The application aims to provide a semantic visual positioning method based on character identification to solve the problem that the use of character identification is only limited to two-dimensional image character extraction results, and only a time-consuming and inefficient character search process containing a large amount of repetition can be performed, and the efficiency and effect of the existing method have a large gap with the actual application requirements.

[0006] To achieve the above object, the application provides the following technical scheme: a semantic visual positioning method based on character identification, the visual positioning method comprising the following steps:

[0007] S1, a preparation step, three-dimensional map construction is performed, and image retrieval and pose estimation are both based on the map;

[0008] S2, an image retrieval step, based on the three-dimensional character map constructed in the previous step, an image retrieval is performed on an input query frame;

[0009] S3, pose estimation, after a matching frame of the current frame is obtained, the relative pose between the two frames needs to be calculated, and at this time, the feature point matching result between the two frames is needed;

[0010] S4, experimental comparison, the data set proposed by TextSLAM is used in the experiment, multiple classical and best methods in the related fields of image retrieval, position recognition, visual positioning and the like are selected for comparison, and the trained model provided by the method based on deep learning is tested.

[0011] Optionally, the S1 further comprises: inputting a group of database frames into a SLAM algorithm or SfM algorithm capable of processing character information to construct a global map containing character semantic information and geometric position information; the character target in the map is a result verified by multiple frames of observation, and compared with the direct character detection result on a single image, the repeated observation and single-frame detection error result are deleted.

[0012] Optionally, the S2 comprises:

[0013] S21, using a character extraction technology to perform character target detection and recognition on a two-dimensional image to obtain all observed characters in the image and the semantic information thereof;

[0014] S22, extracting and calculating the FAST feature points and the BRIEF descriptor thereof in the image, including all feature points in the character region and the non-character region;

[0015] S23, traversing all character observations on the query frame, and for each character observation;

[0016] S24, traversing all character targets in the three-dimensional character map;

[0017] S25, performing similarity comparison between each character observation and each map character target: specifically, using si and s j The similarity between the two strings of character semantic information is defined as follows:

[0018] S26, the threshold is used to screen the character semantic matching result, and the result higher than the threshold is considered as the matched character, and the matching lower than the threshold is deleted, in this step, the selection of the threshold is proposed to calculate the adaptive threshold, and in the case that the current frame exists observation occlusion, etc., the correct matching character target can also be found in the map;

[0019] S27, the matching result screened by the threshold is the map character target matching result of the current character observation, and the database observation frame of all the matched map character targets is extracted from the map, and the frame is counted and accumulated by 1;

[0020] S28, the matching step of the current character observation is ended, and the matching process of the next character observation is started, and the above process is repeated.

[0021] Optionally, the similarity defined by the similarity calculation result value range is (0, 1], wherein |s| is the length of the string, and d(s i ,s j ) is defined as the minimum number of steps required to transform s i into s j , and s(s i ,s j ) is greater, which represents that the two strings are more similar.

[0022] Optionally, the S3 comprises:

[0023] S31, each loop candidate frame is traversed, that is, each pair of query frame-candidate frame matching is performed, and the character matching result of the two frames obtained in the previous step is extracted;

[0024] S32, all character matching is traversed, and character-guided feature point matching is performed, that is, in the two frame image regions in which each pair of character targets is located, BRIEF descriptor is used for feature point matching, so that compared with full image search, the search of feature points is only performed in the reliable region;

[0025] S33, the step S32 is repeated to obtain all character region feature point matching results of the query frame and the current image retrieval candidate frame.

[0026] S34, all character feature point matching results in the previous step are removed, and point matching is performed in the remaining feature points and search regions to obtain the feature point matching results of the non-character region;

[0027] S35, input all feature point matching results, perform SE(3) pose optimization, optimize using re-projection error term, fix the pose of the candidate frame in the optimization, based on the optimization result, delete the feature point matching result that does not meet the result, and obtain the final relative pose calculation result and feature point matching result;

[0028] S36, return to S31 step, and enter the processing flow of the next candidate frame;

[0029] S37, when all image retrieval candidate frames are traversed, a candidate frame with the most feature point matching results with the query frame is selected, and the pose calculation result in the result is used as the final query frame pose result.

[0030] Compared with the prior art, the present application provides a semantic visual positioning method based on text identification, which has the following beneficial effects:

[0031] 1. The semantic visual positioning method based on text identification uses text identification information in the scene as a feature for visual positioning, thereby realizing a true bag-of-words model instead of the visual "bag-of-words" model used in the foregoing work; using semantic information based on text identification for visual positioning can greatly improve the positioning accuracy and efficiency.

[0032] 2. The semantic visual positioning method based on text identification proposes an image similarity comparison flow based on text identification semantic information to obtain image retrieval candidate frames of the query frame; the text identification information at the same position does not change with the change of the observation environment, so the similarity between different image observations can be calculated to judge the similarity between images; the image retrieval accuracy and recall rate based on text identification semantic information are greatly improved.

[0033] 3. The semantic visual positioning method based on text identification proposes a text-guided feature point matching method based on the text matching result obtained in the foregoing step; even if there is a large change in the environment, such as a large angle change, a day-night light change, accurate feature point matching results can still be obtained, and the effect is greatly improved.

[0034] 4. A three-dimensional text map containing text semantic information is proposed for visual positioning; through the TextSLAM algorithm, the three-dimensional text target observed in the database is contained in the map, all text targets are verified to be stable and reliable in multiple frames, outliers and repeated observations are excluded, and the observation frame of each text target is contained in the map. Based on such a map, text matching is only performed in the query frame and the text target in the three-dimensional map; according to the text matching result, the observation frame can be quickly extracted, and the text matching efficiency and matching quality are greatly improved. BRIEF DESCRIPTION OF DRAWINGS

[0035] Figure 1 This is a schematic diagram showing the comparison of the running time of the visual positioning method of the present invention;

[0036] Figure 2 This is a schematic diagram of the image retrieval system of the present invention;

[0037] Figure 3 This is a schematic diagram of the pose estimation system of the present invention;

[0038] Figure 4 This is a comparative effect diagram of the present invention;

[0039] Figure 5 This is a schematic diagram illustrating the comparison of image retrieval results according to the present invention.

[0040] In the picture: Figure 1 The abbreviations "PP", "SP", and "SG" represent Patch2Pix code, SuperPoint code, and SuperGlue code, respectively.

[0041] Figure 4 The first row contains the full image search results, and the second row contains the text-guided feature point matching method proposed in this invention.

[0042] Figure 5 The second and third columns of the second row, the second and fourth columns of the third row, and the second and fourth columns of the fourth row indicate image retrieval errors, while the second, third, fourth, and fifth columns of the first row, the fourth and fifth columns of the second row, the third and third columns of the third row, and the third and fifth columns of the fourth row indicate correct retrieval. Detailed Implementation

[0043] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0044] like Figures 1-5 As shown, the present invention provides a technical solution: a semantic visual localization method based on text identifiers, comprising the following steps:

[0045] S1. Preparation Steps: Construct a 3D map, upon which image retrieval and pose estimation are based. Input a set of database frames into a SLAM or SfM (Struct From Motion) algorithm capable of processing textual information to construct a global map containing textual semantic information and geometric location information. The text targets in the map are the result of multi-frame observation verification, which, compared with the direct text detection results on a single image, removes duplicate observations and single-frame detection errors; at the same time, the map also stores all observation frames for each text target to facilitate efficient subsequent matching steps.

[0046] S2, Image Retrieval Step: Based on the 3D text map constructed in the previous step, input the query frame to perform image retrieval; specifically including the following steps:

[0047] S21. Use text extraction technology to detect and recognize text targets in two-dimensional images to obtain all observed text in the image and its semantic information;

[0048] S22. Extract FAST feature points and their BRIEF descriptors from the computed image, including all feature points in both text and non-text regions.

[0049] S23. Traverse all text observations on the query frame, for each text observation;

[0050] S24. Traverse all text targets in the 3D text map;

[0051] S25. Compare the similarity between each text observation and each map text target: Specifically, use s i and s j The similarity between two strings representing semantic information is defined as follows:

[0052]

[0053] The similarity calculation result has a range of (0, 1], where |s| is the length of the string, and d(s) = 1 / 2. i ,s j ) is defined as s i Transform into s j Minimum number of change steps required, s(s i ,s j The larger the ), the more similar the two strings are;

[0054] S26, using threshold value to filter the text semantic matching result, the result higher than the threshold value is considered as the matched text, and the matching lower than the threshold value is deleted, in this step, the selection of the threshold value is proposed to be calculated by adaptive threshold value, specifically, in the previous step, by traversing all text targets in the map, the similarity calculation result of a certain text observation in the query frame and all text targets is obtained, if the maximum similarity score s max is 1, the threshold value is set to 1, if the maximum similarity score s max <1, the threshold value is set to max(2 / 3s max , 0.35), wherein 0.35 is a minimum protection threshold value, which indicates that at least 1 / 3 consistency is required for successful matching. The above strategy makes our matching adaptive according to the current observation situation, such as the existence of observation occlusion in the current frame, and the correct matching text target can also be found in the map;

[0055] S27, the matching result filtered by the threshold value is the map text target matching result of the current text observation, the database observation frame of all matched map text targets is extracted from the map, and the frame is counted and accumulated by 1;

[0056] S28, the matching step of the current certain text observation is ended, and the matching process of the next text observation is started, and the above process is repeated.

[0057] S29, after the traversal of all text observations in the query frame is ended, the threshold value is used for database frame filtering, and the database frame with a count exceeding the threshold value is retained, if the finally filtered database frame exceeds 10, the top 10 frames are used as image retrieval candidate frames, if less than 10, all database frames are used as image retrieval candidate frames;

[0058] S3, pose estimation, after the matching frame (i.e. image retrieval candidate frame) of the current frame is obtained, the relative pose between the two frames needs to be calculated, at this time, the feature point matching result between the two frames is needed, but the huge appearance change in the environment, such as the huge light change between day and night, the huge observation angle change, etc., all can easily lead to the extremely unstable matching between the feature points, and even the complete failure of the feature point matching, which will lead to the serious error of the pose result. In view of this situation, based on the text matching result obtained in the previous step, the invention proposes a text-guided feature point matching method; specifically including the following steps:

[0059] S31, traversing each loop candidate frame, i.e. matching each pair of query frame-candidate frame, extracting the text matching result of the two frames obtained in the previous step;

[0060] S32, traverse all character matching, character guided feature point matching, that is, in the two frames of image region where each pair of character target is located, feature point matching is performed using BRIEF descriptor, so compared with full image search, the search of feature points is only performed in the reliable region, greatly improving the efficiency and accuracy, and the comparison effect of the method is shown in the following table Figure 4 ;

[0061] S33, repeat S32 to obtain all character region feature point matching results of the query frame and the current image search candidate frame;

[0062] S34, remove all character feature point matching results in the previous step, and perform point matching in the remaining feature points and search regions to obtain feature point matching results of non-character regions. At this point, all feature point matching results of the character region and the non-character region constitute the feature point matching results of the two frames;

[0063] S35, input all feature point matching results, perform SE(3) pose optimization, and optimize using re-projection error term. In the optimization, the pose of the candidate frame is fixed. Based on the optimization result, the feature point matching results that do not meet the result are deleted to obtain the final relative pose calculation result and feature point matching result;

[0064] S36, return to S31 to enter the processing flow of the next candidate frame;

[0065] S37, when all image search candidate frames are traversed, the candidate frame with the most feature point matching results with the query frame is selected, and the pose in the result is used to calculate the final query frame pose result.

[0066] S4, experimental comparison, the data set proposed by TextSLAM is used in the experiment, multiple classical and best methods in the related fields of image search, position recognition, visual positioning, etc. are selected for comparison, and the trained model provided by the method based on deep learning is tested; the test result is shown in the following table Figure 1 ;

[0067] The above describes the application in general, but some modifications or improvements can be made on the basis of the application, which is obvious to those skilled in the art. Therefore, the modifications or improvements without departing from the spirit of the application are within the protection scope of the application.

Claims

1. A semantic visual localization method based on text identifiers, characterized in that: The visual positioning method includes the following steps: S1. Preparation steps: Construct a 3D map, which is used for image retrieval and pose estimation. Input a set of database frames into a SLAM or SfM algorithm that can process text information to construct a global map containing text semantic information and geometric location information. The text targets in the map are the result of multi-frame observation and verification. Compared with the direct text detection results on a single image, duplicate observations and single-frame detection errors have been removed. S2, Image retrieval step: Based on the 3D text map constructed in the previous step, input the query frame to perform image retrieval; S3. Pose estimation: After obtaining the matching frame of the current frame, it is necessary to calculate the relative pose between the two frames. At this time, the feature point matching results between the two frames are required. S31. Traverse each loop candidate frame, that is, each pair of query frames and candidate frames match, and extract the text matching results of the two frames obtained in the previous step. S32. Traverse all text matches and perform text-guided feature point matching. That is, within the two-frame image region where each pair of text targets is located, use the BRIEF descriptor to perform feature point matching. In this way, compared with the full image search, the feature point search is only performed in the reliable region. S33. Repeat step S32 to obtain the matching results of all text region feature points between the query frame and the current image retrieval candidate frame; S4. Experimental Comparison: The experiment uses the dataset proposed by TextSLAM to compare several classic and best methods in the fields of image retrieval, location recognition, and visual positioning. The deep learning-based method is tested using the pre-trained model provided by the method.

2. The semantic visual localization method based on text identifiers according to claim 1, characterized in that: S2 includes: S21. Use text extraction technology to detect and recognize text targets in two-dimensional images to obtain all observed text in the image and its semantic information; S22. Extract FAST feature points and their BRIEF descriptors from the computed image, including all feature points in text regions and non-text regions. S23. Traverse all text observations on the query frame, for each text observation; S24. Traverse all text targets in the 3D text map; S25. Compare the similarity between each text observation and each map text target: Specifically, use... s i and s j The similarity between two strings representing semantic information is defined as follows: 。 3. The semantic visual localization method based on text identifiers according to claim 2, characterized in that: S2 further includes: S26. Use a threshold to filter the text semantic matching results. Results above the threshold are considered to be matched text, and matches below the threshold are deleted. In this step, we propose an adaptive threshold for calculation. Even if there is observation occlusion in the current frame, the correct matching text target can also be found in the map. S27. The matching results after threshold filtering are the map text target matching results of the current text observation. Extract all the database observation frames of the matched map text targets from the map and increment the count of the frame by 1. S28. The matching step for the current text observation ends, and the matching process for the next text observation begins, repeating steps S21-S28.

4. The semantic visual localization method based on text identifiers according to claim 3, characterized in that: The similarity calculation result defined thereon has a range of (0, 1], where The length of the string. Define as Transform into Minimum number of change steps required The larger the value, the more similar the two strings are.

5. The semantic visual localization method based on text identifiers according to claim 4, characterized in that: S3 further includes: S34. Remove all text feature point matching results from the previous step, and perform point matching on the remaining feature points and the search area to obtain the feature point matching results for the non-text area. S35. Input all feature point matching results, perform SE (3) pose optimization, optimize using reprojection error term, fix the pose of candidate frames during optimization, delete feature point matching results that do not conform to the results based on the optimization results, and obtain the final relative pose calculation results and feature point matching results. S36. Return to step S31 and proceed to the processing flow of the next candidate frame; S37. After traversing all image retrieval candidate frames, select the candidate frame with the most feature point matching results with the query frame, and use the pose in the result to calculate the final query frame pose result.

Images