Positioning method, device and terminal equipment based on semantic information

Abstract

The present application provides a positioning method, device and terminal device based on semantic information, which includes: semantically segmenting the acquired RGBD image to obtain the semantic label of each pixel; reconstructing the point cloud based on the RGBD image, and mapping the semantic label of each pixel to the reconstructed point cloud to obtain the semantic point cloud of the current frame with semantic labels and the semantic map of the three-dimensional scene; generating two-dimensional discrete semantic feature points for each three-dimensional semantic point cloud in the current frame and the semantic map, respectively, to obtain the corresponding two-dimensional semantic feature point image, and then performing three-dimensional semantic feature description on each feature point in the two-dimensional semantic feature point image; based on the three-dimensional semantic feature description, feature matching is performed on all feature points in the current frame and all feature points in the semantic map to obtain positioning information. By introducing semantic features, this method can effectively improve the feature matching efficiency during positioning and reduce the probability of mismatching.

Description

Technical Field

[0001] The present application relates to the field of positioning technology, and in particular to a positioning method, apparatus and terminal device based on semantic information. Background Art

[0002] The traditional positioning method of the simultaneous localization and mapping (SLAM) system lacks semantic information, so it completely relies on geometric features and visual features in the positioning process. In the absence of semantic information, it is necessary to traverse the entire map to search for the position that best matches the current position information. Therefore, the positioning algorithm is relatively time-consuming and prone to mismatching. Especially in visual SLAM, the positioning method based on visual features is easily affected by changes in lighting and scenes, and is also extremely sensitive to dynamic scenes, which affects the positioning accuracy. Summary of the invention

[0003] In view of this, the embodiments of the present application provide a positioning method, apparatus and terminal device based on semantic information, which can effectively solve the positioning problems in dynamic changes and lighting changes that cannot be solved in traditional SLAM systems.

[0004] In a first aspect, an embodiment of the present application provides a positioning method based on semantic information, including:

[0005] Perform semantic segmentation on the acquired depth image to obtain the semantic label of each pixel;

[0006] Reconstructing a point cloud based on the RGBD image, and mapping the semantic label of each pixel to the reconstructed point cloud, to obtain a current frame semantic point cloud with semantic labels and a three-dimensional scene semantic map respectively;

[0007] Generating two-dimensional discrete semantic feature points for each three-dimensional semantic point cloud in the current frame and the semantic map to obtain corresponding two-dimensional semantic feature point images, and then performing three-dimensional semantic feature description on each feature point in the two-dimensional semantic feature point images;

[0008] Based on the three-dimensional semantic feature description, feature matching is performed on all feature points in the current frame and all feature points in the semantic map to obtain positioning information.

[0009] In some embodiments, after obtaining the positioning information, the method further includes:

[0010] The semantic feature point map constructed based on the two-dimensional semantic feature point image is divided into a preset number of sub-maps, wherein there is an overlapping area between each sub-map that matches the observation range of the current frame; then, a partial semantic point cloud in the current frame and a target semantic point cloud with the same semantic label in the sub-map are extracted, and the semantic point clouds are aligned using an iterative closest point algorithm to obtain high-precision positioning information.

[0011] In some embodiments, the generating of two-dimensional discrete semantic feature points for each three-dimensional semantic point cloud in the current frame and the semantic map to obtain corresponding two-dimensional semantic feature point images includes:

[0012] Projecting each 3D semantic point cloud in the current frame and the semantic map onto a 2D plane, and calculating the 2D feature image coordinates corresponding to each 3D semantic point cloud;

[0013] The number of three-dimensional point clouds of different semantic categories corresponding to each of the two-dimensional feature image coordinates is counted, and the semantic label of the corresponding semantic category with the largest number is used as the pixel value of the two-dimensional feature point, until all three-dimensional semantic point clouds are traversed to generate the corresponding two-dimensional semantic feature point image.

[0014] In some embodiments, the calculation to obtain the two-dimensional feature image coordinates corresponding to each three-dimensional semantic point cloud includes:

[0015] The three-dimensional semantic point cloud is transformed from the camera coordinate system to a coordinate system in which the ground is an XY coordinate plane and the Z axis is perpendicular to the ground and upward, and then the minimum and maximum values ​​of each coordinate in the three-dimensional semantic point cloud are respectively solved;

[0016] The resolution of the two-dimensional discretized image is calculated according to the resolution of the two-dimensional discretization, the minimum value and the maximum value, and the two-dimensional feature image coordinates corresponding to the three-dimensional semantic point cloud at the image resolution are calculated.

[0017] In some embodiments, the performing three-dimensional semantic feature description on each feature point in the two-dimensional semantic feature point image includes:

[0018] Based on a proximity search algorithm, a plurality of neighboring feature points within a preset search radius corresponding to each feature point in the two-dimensional semantic feature point image are searched, and then all three-dimensional semantic point clouds corresponding to the plurality of neighboring feature points are extracted as local semantic point clouds;

[0019] The local semantic point cloud is divided into different circular areas, and then the number of semantic categories in different areas is searched and counted to construct a semantic histogram of the corresponding area, and then the semantic histograms of each area are connected to obtain a three-dimensional semantic feature description of the feature point.

[0020] In some embodiments, the area radius of the local semantic point cloud is R, and the number of divided rings is N. R , the search rules for the number of semantic categories in different regions are as follows:

[0021] The search radius is R / N respectively R The circular area with an inner radius of k*R / N R And the outer diameter is (k+1)*R / N R The annular area, where k is [1, ..., N R -1] until all areas have been searched.

[0022] In some embodiments, performing feature matching on all feature points in the current frame and all feature points in the semantic map includes:

[0023] The current frame and each feature point with the same semantic label in the semantic map are matched to obtain a corresponding number of matching pairs, and the matching pairs are used to obtain positioning information in a two-dimensional space.

[0024] In some embodiments, performing feature matching on all feature points in the current frame and all feature points in the semantic map includes:

[0025] Dividing the semantic feature point map constructed based on the two-dimensional semantic feature point image into a preset number of sub-maps, wherein there are overlapping areas between the sub-maps that conform to the observation range of the current frame;

[0026] The feature description of each submap is extracted respectively to perform feature matching on all feature points in the current frame with each submap to obtain a number of matching pairs corresponding to each submap, and all submaps are screened based on the number of matching pairs to obtain the most matching submap, and the most matching submap is used to obtain positioning information in a two-dimensional space.

[0027] In some embodiments, the filtering of all sub-maps based on the plurality of matching pairs includes:

[0028] The matching pairs are randomly selected to solve the corresponding transformation matrix, and then the inliers are solved for all matching pairs, and the submap with the largest number of inliers is taken as the best matching submap.

[0029] In a second aspect, an embodiment of the present application provides a positioning device based on semantic information, including:

[0030] The image semantic segmentation module is used to perform semantic segmentation on the acquired depth image to obtain the semantic label of each pixel;

[0031] A semantic map construction module, used to reconstruct a point cloud based on the RGBD image, and map the semantic label of each pixel to the reconstructed point cloud, to obtain a current frame semantic point cloud with semantic labels and a three-dimensional scene semantic map;

[0032] A semantic feature description module is used to generate two-dimensional discrete semantic feature points for each three-dimensional semantic point cloud in the current frame and the semantic map, obtain corresponding two-dimensional semantic feature point images, and then perform three-dimensional semantic feature description on each feature point in the two-dimensional semantic feature point images;

[0033] The feature matching module is used to perform feature matching on all feature points in the current frame and all feature points in the semantic map based on the three-dimensional semantic feature description to obtain positioning information.

[0034] In a third aspect, an embodiment of the present application provides a terminal device, comprising a processor and a memory, wherein the memory stores a computer program, and the processor is used to execute the computer program to implement the above-mentioned positioning method based on semantic information.

[0035] In a fourth aspect, an embodiment of the present application provides a readable storage medium storing a computer program, which, when executed on a processor, implements the above-mentioned positioning method based on semantic information.

[0036] The embodiments of the present application have the following beneficial effects:

[0037] The embodiment of the present application proposes a positioning method based on semantic information. The method first performs pixel-level semantic segmentation, and then uses the results of semantic segmentation to perceive the scene in real time, clusters static objects, dynamic objects, etc. and corresponding semantic labels, effectively eliminates the interference of dynamic objects on positioning, and can also avoid the problem of sensitivity to light in traditional methods at the same time; then, after obtaining the semantic point cloud of the current frame with semantic labels and the three-dimensional scene semantic map, first generate two-dimensional discretized semantic feature points, and then use three-dimensional geometric information to describe the three-dimensional semantic features of these feature points, and finally perform semantic feature matching to achieve positioning. By effectively fusing the semantic segmentation results and three-dimensional geometric information, a three-dimensional semantic feature description based on two-dimensional constraints is achieved, and then the semantic features are used for efficient feature matching and positioning, which can effectively improve the efficiency of feature matching and reduce the probability of mismatching. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0039] Figure 1 A schematic diagram of the structure of a terminal device according to an embodiment of the present application is shown;

[0040] Figure 2 A first flow chart of a positioning method based on semantic information according to an embodiment of the present application is shown;

[0041] Figure 3 A flowchart of generating semantic feature points of a positioning method based on semantic information according to an embodiment of the present application is shown;

[0042] Figure 4 A schematic diagram of a two-dimensional feature image of a positioning method based on semantic information according to an embodiment of the present application is shown;

[0043] Figure 5 A flowchart of three-dimensional semantic feature description of a positioning method based on semantic information according to an embodiment of the present application is shown;

[0044] Figure 6 A schematic diagram of a semantic ring of a positioning method based on semantic information according to an embodiment of the present application is shown;

[0045] Figure 7 A feature matching flow chart of a positioning method based on semantic information according to an embodiment of the present application is shown;

[0046] Figure 8 A second flow chart of the positioning method based on semantic information according to an embodiment of the present application is shown;

[0047] Fig. 9 A structural schematic diagram of a positioning device based on semantic information according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0048] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0049] The components of the embodiments of the present application generally described and shown in the drawings herein may be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0050] Hereinafter, the terms "including", "having" and their cognates that can be used in various embodiments of the present application are intended only to indicate specific features, numbers, steps, operations, elements, components or a combination of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or a combination of the foregoing items or increasing the possibility of one or more features, numbers, steps, operations, elements, components or a combination of the foregoing items. In addition, the terms "first", "second", "third" and the like are only used to distinguish descriptions and cannot be understood as indicating or implying relative importance.

[0051] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present application belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present application.

[0052] In conjunction with the accompanying drawings, some embodiments of the present application are described in detail below. In the absence of conflict, the following embodiments and features in the embodiments can be combined with each other.

[0053] Please refer to Figure 1 , which is a schematic diagram of the structure of the terminal device 10 proposed in the embodiment of the present application. For example, the terminal device 10 may include but is not limited to a robot, an unmanned vehicle, etc. Exemplarily, the terminal device 10 includes a memory 11 and a processor 12, and of course may also include other perception units, such as a depth camera, a laser radar, etc., which can be set according to actual needs. Among them, the memory 11 stores a computer program, and the processor 12 is used to execute the computer program to implement the functions of each module in the positioning method or device based on semantic information in the embodiment of the present application, thereby effectively solving the positioning problems in the traditional SLAM system that cannot solve the dynamic changes and illumination changes.

[0054] In this embodiment, the memory 11 may be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc. The memory 11 is used to store a computer program, and the processor 12 may execute the computer program accordingly after receiving an execution instruction.

[0055] In this embodiment, the processor 12 may be an integrated circuit chip with signal processing capability. The processor 12 may be a general-purpose processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or at least one of other programmable logic devices, discrete gates or transistor logic devices, and discrete hardware components. The general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc., and may implement or execute the disclosed methods, steps, and logic block diagrams in the embodiments of the present application.

[0056] Based on the terminal device 10 of the above structure, the embodiment of the present application proposes a positioning method based on semantic information, which can be applied in some SLAM positioning scenarios, especially visual SLAM, and can well solve the positioning problem of the terminal device in situations such as dynamic scene changes and lighting changes.

[0057] Please refer to Figure 2 The positioning method based on semantic information includes steps S110 to S140:

[0058] S110, performing semantic segmentation on the acquired RGBD image to obtain a semantic label for each pixel.

[0059] Semantic segmentation and semantic mapping are the basis of semantic localization. Before semantic feature description and semantic localization, semantic segmentation and semantic mapping need to be performed first. Exemplarily, a depth camera mounted on a terminal device such as a robot or an unmanned vehicle can be used to first obtain an RGBD image captured by the depth camera. Optionally, some preprocessing operations can be performed on the obtained RGBD image; then, semantic segmentation of the RGBD image is performed based on deep learning to obtain a semantic label for each pixel, that is, a pixel-level semantic label. It can be understood that deep learning can be used to pre-train semantic segmentation on sample images that have been annotated with semantic labels to obtain a corresponding semantic segmentation model.

[0060] Among them, RGBD images include two types of images: one is a three-channel color image of R (Red), G (Green), and B (Blue), and the other is a depth image, which is similar to a grayscale image, in which each pixel value reflects the distance between the sensor and the object. Usually, the pixels in the RGB image and the depth image are aligned, that is, they have a one-to-one correspondence.

[0061] It can be understood that semantic segmentation can be used to perceive the scene, distinguish and cluster dynamic objects and static objects in the scene, and effectively eliminate the interference of dynamic objects on positioning. In addition, semantic segmentation is more robust to lighting, and semantic positioning based on semantic segmentation results can also avoid the problem of sensitivity to lighting in traditional methods.

[0062] S120, reconstructing the point cloud based on the RGBD image, and mapping the semantic label of each pixel to the reconstructed point cloud, to obtain a current frame semantic point cloud with semantic labels and a three-dimensional scene semantic map.

[0063] Exemplarily, the point cloud of a single frame and the entire three-dimensional scene can be reconstructed based on the depth image and camera internal parameters in the RGBD image, that is, the point cloud of a single frame and the point cloud map of the entire scene are reconstructed respectively; at the same time, during the reconstruction process, the semantic label of each pixel is also mapped to the reconstructed dense point cloud map to obtain the current frame point cloud with semantic labels (i.e., the current frame semantic point cloud) and the scene point cloud map with semantic labels (i.e., the three-dimensional scene semantic map), so that the semantic mapping is completed. It can be understood that since the three-dimensional point clouds in the current frame semantic point cloud and the three-dimensional scene semantic map have semantic labels, they are also called three-dimensional semantic point clouds.

[0064] S130, generating two-dimensional discrete semantic feature points for each three-dimensional semantic point cloud in the current frame and the semantic map, obtaining corresponding two-dimensional semantic feature point images, and then performing three-dimensional semantic feature description on each feature point in the two-dimensional semantic feature point images.

[0065] Considering that feature matching in three-dimensional space is more likely to cause matching failure than in two-dimensional space, and there are fewer internal points after screening by the sampling consistency algorithm, this embodiment proposes to first perform feature description and matching in two-dimensional space, and then combine three-dimensional geometric information to perform three-dimensional semantic description of these feature points.

[0066] The generation of two-dimensional discrete semantic feature points refers to performing two-dimensional discretization processing on the reconstructed three-dimensional point cloud to obtain feature points with semantic label information for constructing a two-dimensional image. Figure 3 As shown in Figure 1, the process of generating two-dimensional discrete semantic feature points includes:

[0067] S210, projecting each 3D semantic point cloud in the current frame and the semantic map onto a 2D plane, and calculating and obtaining the 2D feature image coordinates corresponding to each 3D semantic point cloud.

[0068] Exemplarily, each 3D point cloud in the current frame and the semantic map can be transformed from the camera coordinate system to a coordinate system where the ground is the XY coordinate plane and the Z axis is perpendicular to the ground and upward, and then the minimum and maximum values ​​of each coordinate in the 3D semantic point cloud are solved respectively. It can be understood that the projection plane of the 3D point cloud is not limited to the ground, but can also be other planes set according to the actual scene. This is just an example. Then, according to the set 2D discretization resolution, the minimum value and the maximum value, the image resolution after 2D discretization is calculated, and the 2D feature image coordinates corresponding to the 3D semantic point cloud at the image resolution are calculated.

[0069] For example, the minimum and maximum values ​​of each coordinate of the input 3D semantic point cloud are represented as P min =(x min ,y min ,z min ) and P max =(x max ,y max ,z max ), let the resolution of two-dimensional discretization be r, then the corresponding image resolution after two-dimensional discretization is:

[0070]

[0071] In the formula, the symbol Indicates rounding up, I w and I h Represents the width and height of the image size respectively.

[0072] Therefore, according to the x-coordinate and y-coordinate values ​​of each point in the semantic point cloud, the two-dimensional image coordinates corresponding to each three-dimensional point can be obtained, that is,

[0073]

[0074] Where (u, v) represents the coordinates of the two-dimensional feature image, and (x, y) represents the coordinate values ​​of the three-dimensional semantic point cloud on the x-axis and y-axis.

[0075] For example, Figure 4 Shown is a two-dimensional feature image composed of a number of two-dimensional feature points, wherein the two-dimensional feature image has different semantic categories represented by different colors (colors not shown).

[0076] S220, counting the number of three-dimensional point clouds of different semantic categories corresponding to each two-dimensional feature image coordinate, and taking the semantic label of the corresponding semantic category with the largest number as the pixel value of the two-dimensional feature point, until all three-dimensional semantic point clouds are traversed to generate the corresponding two-dimensional semantic feature point image.

[0077] Among them, a three-dimensional semantic point cloud usually includes multiple semantic categories, which are recorded as c here. Exemplarily, after the two-dimensional feature image coordinates corresponding to each point cloud are obtained, the number of semantic three-dimensional points of different categories corresponding to each image coordinate is counted, and then the corresponding semantic label with the largest number of semantic category points is assigned to the two-dimensional feature point (also called pixel point) under the current image coordinate. The pixel value of the feature point is the assigned semantic category label. After traversing all three-dimensional semantic point clouds, the corresponding two-dimensional semantic feature point image can be obtained.

[0078] Next, after generating the two-dimensional semantic feature points, this embodiment will further combine the three-dimensional spatial geometric features to perform three-dimensional semantic feature description of the semantic feature points. By simultaneously considering the three-dimensional spatial geometric information and semantic category information of the features, the robustness of the semantic feature matching can be improved.

[0079] In one embodiment, if Figure 5 As shown, a three-dimensional semantic feature description is performed on each feature point in the two-dimensional semantic feature point image, including:

[0080] S230, searching for a number of neighboring feature points within a preset search radius corresponding to each feature point in the two-dimensional semantic feature point image based on a neighboring search algorithm, and then extracting all three-dimensional semantic point clouds corresponding to the number of neighboring feature points as local semantic point clouds.

[0081] Exemplarily, in the process of generating the above-mentioned two-dimensional semantic feature points, the correspondence between each three-dimensional point and the two-dimensional feature point can be recorded at the same time. In order to realize the three-dimensional semantic feature description, the local semantic features at each feature point can be first extracted based on the three-dimensional semantic point cloud. For example, in the local feature description process of each two-dimensional feature point, a proximity search algorithm such as a two-dimensional KD tree can be used to search for neighboring feature points within a certain search radius corresponding to each feature point, and then all three-dimensional semantic point clouds corresponding to these neighboring feature points are extracted, wherein these extracted semantic point clouds carry different semantic labels. It can be understood that the search radius can be set according to actual needs and is not limited here.

[0082] S240, dividing the local semantic point cloud into different circular areas, then searching and counting the number of semantic categories in different areas to construct a semantic histogram of the corresponding area, and then connecting the semantic histograms of each area to obtain a three-dimensional semantic feature description of the feature point.

[0083] In order to effectively combine local geometric information and semantic information, the local semantic point cloud obtained in this embodiment is divided into different annular regions to ensure the rotation invariance of the local feature description, which is convenient for feature matching for arbitrary rotation transformation. Exemplarily, a neighboring search algorithm of a three-dimensional point cloud such as a KD tree search algorithm can also be used to perform annular division to obtain different multiple regions, and then perform semantic category search and number statistics on these regions.

[0084] For example, the radius of a local semantic point cloud is R, and the number of divided rings is N. R In one embodiment, the search rule for the number of semantic categories in different regions may be: search radius R / N respectively. R The circular area with an inner radius of k*R / N R And the outer diameter is (k+1)*R / N R The annular area, where k is [1, ..., N R -1] until all areas are searched. In this way, different areas can be divided and then searched at the same time.

[0085] As another optional search scheme, you can first search for a radius of R / N R The number of points of different semantic categories contained in the circular area is counted, and the histogram of the semantic categories in the area is constructed. Then, the searched radius is R / N R The point cloud in the circular area of ​​​​the radius of 2R / N is removed from the local semantic point cloud. R Perform area search and count the number of semantic points, and so on, until the radius of N is counted. R *R / N RThe corresponding area.

[0086] Finally, the above N R The semantic histograms of the regional statistics are connected to form the three-dimensional semantic feature description of the semantic feature point, such as Figure 6 The figure shows the semantic point cloud corresponding to a circle with different radius. It can be understood that the feature dimension of each semantic feature point is N R *c.

[0087] S140, based on the three-dimensional semantic feature description, feature matching is performed on all feature points in the current frame and all feature points in the semantic map to obtain positioning information.

[0088] Among them, feature matching is to match the description of all semantic feature points in the current frame with the description of the corresponding feature points in the map. For the matching process of the above step S140, for example, in one embodiment, each feature point with the same semantic label in the current frame and the semantic map can be directly matched to obtain a corresponding number of matching pairs, wherein the obtained matching pairs can be used to determine the current posture, and then obtain its positioning information in the two-dimensional space. It can be understood that the method of directly matching the same semantic label can improve the matching speed on the one hand, and on the other hand, it is also conducive to improving the matching success rate.

[0089] In addition, considering some scenarios, if the feature points of the current frame are directly matched with all the feature points of the map, due to the feature similarity, the number of matching points that are actually successfully matched may be relatively small, which may lead to low positioning accuracy or positioning errors. For this reason, a sub-map-based registration scheme is proposed here.

[0090] As an alternative, Figure 7 As shown, the matching process of step S140 includes:

[0091] S310, dividing the semantic feature point map constructed based on the two-dimensional semantic feature point image into a preset number of sub-maps, wherein there is an overlapping area between each sub-map that conforms to the observation range of the current frame. It can be understood that after obtaining the two-dimensional semantic feature point image, the map information of the entire two-dimensional semantic feature point can be obtained.

[0092] Exemplarily, by dividing the entire two-dimensional semantic feature point map into multiple sub-maps, it should be ensured that there are overlapping areas between the sub-maps at the same time during the division, so that any position of the current frame relative to the map can be matched with a sub-map. Among them, the area of ​​the overlapping area can be set accordingly according to the observation range of the current frame, for example, equal to the observation range of the current frame, etc. In some cases, it can also be less than or greater than, etc., which can be adjusted according to the actual scene.

[0093] S320, extracting feature descriptions of each submap respectively, performing feature matching on all feature points in the current frame and each submap, obtaining a number of matching pairs corresponding to each submap, and screening all submaps based on the number of matching pairs to obtain the most matching submap, and the most matching submap is used to obtain positioning information in the two-dimensional space.

[0094] After the sub-maps are divided, the feature descriptions of each sub-map can be extracted respectively, and then the current frame is matched with each sub-map. For example, suppose the number of feature points in the current frame is N source , there are N submaps target feature points, the dimension of each feature descriptor is N R *c, using the nearest neighbor open source library such as FLANN, first establish N R *c-dimensional KD tree, and then through k-nearest neighbor search, search in the submap to get the two features of the nearest neighbor of each feature point in the current frame. If the ratio between the Euclidean distance of the nearest neighbor and the second nearest neighbor feature is less than the preset threshold δ, it is considered to be a matching pair with the nearest neighbor feature, otherwise the matching pair is not retained. In this way, a certain number of matching pairs can be obtained between the current frame and each submap.

[0095] However, since these matching relationships may be wrong, it is also necessary to select the sub-map that best matches. For example, when screening sub-maps, the corresponding transformation matrix can be solved by continuously and randomly selecting the matching pairs, and then the inliers are solved for all matching pairs. The sub-map with the largest number of inliers is used as the best matching sub-map. This process is also called the RANSAC algorithm. Finally, the two-dimensional rotation and translation matrices are calculated based on the coordinates of the matching points to further calculate the positioning information. It can be understood that this process ensures that the relative distance between matching pairs is not too far. Through this spatial constraint, the probability of successful matching can be effectively improved, and the robustness of feature matching can be improved.

[0096] At this point, the above process completes the positioning in two-dimensional space. The embodiment of the present application first performs pixel-level semantic segmentation, and then, after obtaining the semantic point cloud of the current frame with semantic labels and the three-dimensional scene semantic map, first generates two-dimensional discretized semantic feature points and then uses three-dimensional geometric information to perform three-dimensional semantic feature description on these feature points, and finally performs semantic feature matching to achieve positioning. By effectively fusing the semantic segmentation results and three-dimensional geometric information, a three-dimensional semantic feature description based on two-dimensional constraints is achieved, and then the semantic features are used for efficient feature matching and positioning, which can effectively improve the feature matching efficiency and reduce the probability of mismatching.

[0097] As an optional solution, considering that the resolution of the two-dimensional semantic feature point map is low, the positioning accuracy is relatively low, and no ground position correction is performed in the Z-axis direction. Therefore, the embodiment of the present application also proposes to perform high-precision three-dimensional positioning again on the basis of two-dimensional positioning.

[0098] Please refer to Figure 8 After step S140, the positioning method based on semantic information further includes:

[0099] S150, divide the semantic feature point map constructed based on the two-dimensional semantic feature point image into a preset number of sub-maps. Among them, there are overlapping areas between each sub-map that match the observation range of the current frame. It can be understood that this step is the same as the above-mentioned step S310, and it will not be repeated here. If the semantic feature point map has been divided before, it can be adopted here. Then, extract the partial semantic point cloud in the current frame and the target semantic point cloud with the same semantic label in the sub-map, and then use the iterative closest point algorithm to perform semantic point cloud registration to obtain high-precision positioning information.

[0100] Demonstratively, after completing the two-dimensional positioning, it is used as the initial position to extract part of the semantic point cloud in the current frame and the sub-map point cloud of the same semantic category again, and then the two high-precision point clouds are aligned using the iterative closest point algorithm to achieve the final positioning.

[0101] Optionally, considering the efficiency and robustness of the precise positioning algorithm, after extracting the high-precision point cloud, the extracted point cloud can be downsampled using the voxel filtering method, and the downsampled point cloud can be used for the above-mentioned registration operation, which can effectively reduce the sensitivity of the iterative closest point algorithm to the initial position, and at the same time significantly improve the registration speed. Although the point cloud is downsampled in this process, the relative position relationship between the point clouds constrains the actual object geometry, so the impact on the precision of precise positioning is limited. Under the premise of appropriate downsampling, the precision of precise positioning can still be guaranteed.

[0102] Please refer to Fig. 9 Based on the method of the above embodiment, this embodiment proposes a positioning device 100 based on semantic information. Exemplarily, the positioning device 100 based on semantic information includes:

[0103] The image semantic segmentation module 110 is used to perform semantic segmentation on the acquired RGBD image to obtain a semantic label for each pixel;

[0104] The semantic map construction module 120 is used to reconstruct the point cloud based on the depth image and map the semantic label of each pixel to the reconstructed point cloud to obtain the semantic point cloud of the current frame with the semantic label and the semantic map of the three-dimensional scene;

[0105] The semantic feature description module 130 is used to generate two-dimensional discrete semantic feature points for each three-dimensional semantic point cloud in the current frame and the semantic map, obtain corresponding two-dimensional semantic feature point images, and then perform three-dimensional semantic feature description on each feature point in the two-dimensional semantic feature point images;

[0106] The matching and positioning module 140 is used to perform feature matching on all feature points in the current frame and all feature points in the semantic map based on the three-dimensional semantic feature description to obtain positioning information.

[0107] Furthermore, the matching and positioning module 140 can also be used to divide the semantic feature point map constructed based on the two-dimensional semantic feature point image into a preset number of sub-maps, wherein there is an overlapping area between each sub-map that matches the observation range of the current frame; then, extract the partial semantic point cloud in the current frame and the target semantic point cloud with the same semantic label in the sub-map, and then use the iterative closest point algorithm to align the semantic point clouds to obtain high-precision positioning information.

[0108] It can be understood that the device of this embodiment corresponds to the method of the above embodiment, and the options in the above embodiment are also applicable to this embodiment, so they will not be described repeatedly here.

[0109] The present application also provides a readable storage medium for storing the computer program used in the above-mentioned terminal device.

[0110] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and structure diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in an alternative implementation, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the structure diagram and / or the flow diagram, and the combination of boxes in the structure diagram and / or the flow diagram, can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0111] In addition, the functional modules or units in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0112] If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a smart phone, a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0113] The above description is only a specific implementation manner of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A positioning method based on semantic information, characterized in that: include: Perform semantic segmentation on the acquired RGBD image to obtain the semantic label of each pixel; Reconstructing a point cloud based on the RGBD image, and mapping the semantic label of each pixel to the reconstructed point cloud, to obtain a current frame semantic point cloud with semantic labels and a three-dimensional scene semantic map respectively; Projecting each 3D semantic point cloud in the current frame and the semantic map onto a 2D plane, calculating the 2D feature image coordinates corresponding to each 3D semantic point cloud; counting the number of 3D point clouds of different semantic categories corresponding to each 2D feature image coordinate, and taking the semantic label of the semantic category with the largest number as the pixel value of the 2D feature point, until all 3D semantic point clouds are traversed to generate the corresponding 2D semantic feature point image; Based on the proximity search algorithm, a number of neighboring feature points corresponding to each feature point in the two-dimensional semantic feature point image are searched, and then all three-dimensional semantic point clouds corresponding to the number of neighboring feature points are extracted as local semantic point clouds; the local semantic point cloud is divided into different annular areas, and the number of semantic categories in different areas is searched and counted to construct a semantic histogram of the corresponding area, and the semantic histograms of each area are connected to obtain a three-dimensional semantic feature description of each feature point; Based on the three-dimensional semantic feature description, feature matching is performed on all feature points in the current frame and all feature points in the semantic map to obtain positioning information.

2. The positioning method based on semantic information according to claim 1, characterized in that: After obtaining the positioning information, the method further includes: The semantic feature point map constructed based on the two-dimensional semantic feature point image is divided into a preset number of sub-maps, wherein there is an overlapping area between each sub-map that matches the observation range of the current frame; then, a partial semantic point cloud in the current frame and a target semantic point cloud with the same semantic label in the sub-map are extracted, and the semantic point clouds are aligned using an iterative closest point algorithm to obtain high-precision positioning information.

3. The positioning method based on semantic information according to claim 1, characterized in that: The calculation to obtain the two-dimensional feature image coordinates corresponding to each three-dimensional semantic point cloud includes: The three-dimensional semantic point cloud is transformed from the camera coordinate system to a coordinate system in which the ground is an XY coordinate plane and the Z axis is perpendicular to the ground and upward, and then the minimum and maximum values ​​of each coordinate in the three-dimensional semantic point cloud are respectively solved; The resolution of the two-dimensional discretized image is calculated according to the resolution of the two-dimensional discretization, the minimum value and the maximum value, and the two-dimensional feature image coordinates corresponding to the three-dimensional semantic point cloud at the image resolution are calculated.

4. The positioning method based on semantic information according to claim 1, characterized in that: The method of searching for a number of neighboring feature points corresponding to each feature point in the two-dimensional semantic feature point image based on a proximity search algorithm includes: A plurality of neighboring feature points within a preset search radius corresponding to each feature point in the two-dimensional semantic feature point image are searched.

5. The positioning method based on semantic information according to claim 1, characterized in that: The area radius of the local semantic point cloud is R, and the number of divided rings is N R , the search rules for the number of semantic categories in different regions are as follows: The search radius is R / N respectively R The circular area with an inner radius of k*R / N R And the outer diameter is (k+1)*R / N R The annular area, where k is [1, ..., N R -1] until all areas have been searched.

6. The positioning method based on semantic information according to claim 1, characterized in that: The feature matching of all feature points in the current frame and all feature points in the semantic map comprises: The current frame and each feature point with the same semantic label in the semantic map are matched to obtain a corresponding number of matching pairs, and the matching pairs are used to obtain positioning information in a two-dimensional space.

7. The positioning method based on semantic information according to claim 1, characterized in that: The feature matching of all feature points in the current frame and all feature points in the semantic map comprises: Dividing the semantic feature point map constructed based on the two-dimensional semantic feature point image into a preset number of sub-maps, wherein there are overlapping areas between the sub-maps that conform to the observation range of the current frame; The feature description of each submap is extracted respectively to perform feature matching on all feature points in the current frame with each submap to obtain a number of matching pairs corresponding to each submap, and all submaps are screened based on the number of matching pairs to obtain the most matching submap, and the most matching submap is used to obtain positioning information in a two-dimensional space.

8. The positioning method based on semantic information according to claim 7, characterized in that: The screening of all sub-maps based on the plurality of matching pairs includes: The matching pairs are randomly selected to solve the corresponding transformation matrix, and then the inliers are solved for all matching pairs, and the submap with the largest number of inliers is taken as the best matching submap.

9. A positioning device based on semantic information, characterized in that: include: The image semantic segmentation module is used to perform semantic segmentation on the acquired RGBD image to obtain the semantic label of each pixel; A semantic map construction module, used to reconstruct a point cloud based on the RGBD image, and map the semantic label of each pixel to the reconstructed point cloud, to obtain a current frame semantic point cloud with semantic labels and a three-dimensional scene semantic map; A semantic feature description module is used to project each 3D semantic point cloud in the current frame and the semantic map onto a 2D plane, calculate the 2D feature image coordinates corresponding to each 3D semantic point cloud; count the number of 3D point clouds of different semantic categories corresponding to each 2D feature image coordinate, and use the semantic label of the largest number of corresponding semantic categories as the pixel value of the 2D feature point, until all 3D semantic point clouds are traversed to generate the corresponding 2D semantic feature point image; The semantic feature description module is further used to search for a number of neighboring feature points corresponding to each feature point in the two-dimensional semantic feature point image based on a neighboring search algorithm, and then extract all three-dimensional semantic point clouds corresponding to the several neighboring feature points as local semantic point clouds; divide the local semantic point cloud into different annular areas, search and count the number of semantic categories in different areas to construct a semantic histogram of the corresponding area, connect the semantic histograms of each area, and obtain a three-dimensional semantic feature description of each feature point; The feature matching module is used to perform feature matching on all feature points in the current frame and all feature points in the semantic map based on the three-dimensional semantic feature description to obtain positioning information.

10. A terminal device, characterized in that: The terminal device comprises a processor and a memory, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the semantic information-based positioning method according to any one of claims 1 to 8.

11. The terminal device according to claim 10, characterized in that: The terminal device is a robot or an unmanned driving device.

12. A readable storage medium, characterized in that: The device stores a computer program, which, when executed on a processor, implements the positioning method based on semantic information according to any one of claims 1 to 8.

Images