A method and system for constructing vectorized feature sets for indoor scene point cloud models

Abstract

This invention discloses a method and system for constructing a vectorized feature set of an indoor scene point cloud model. The method includes acquiring point cloud data, visible light images from different directions, and depth maps for the indoor scene; performing target object segmentation on the point cloud data based on semantics, and image segmentation on the visible light images and depth maps based on edges; extracting 3D features from enclosed regions of the point cloud, visible light images, and depth maps based on the segmentation results; and then fusing them to generate a vectorized representation of the spatial 3D visual data of the indoor scene. This invention is simple to implement and can achieve a complete expression of multi-dimensional scene data based on point cloud, visible light, and depth map data. Due to the use of a graph structure for data vectorization and fusion, the computational load can be effectively reduced, enabling efficient and low-cost construction of a vectorized feature set for an indoor scene point cloud model.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically to a method and system for constructing a vectorized feature set of an indoor scene point cloud model. Background Technology

[0002] A robot's visual localization capability relies on the efficiency of feature retrieval and matching accuracy of 3D maps of real-world scenes. High-precision dense 3D point cloud maps acquired by LiDAR can accurately reflect the real-world situation, but their large data volume and complex structure lead to time-consuming and inefficient matching of corresponding features between images and point cloud maps during visual localization, failing to meet real-time visual localization requirements. Therefore, to address the needs of real-time visual localization, it is necessary to research lightweight storage methods for high-precision dense point cloud maps, focusing on multi-granularity and multi-level feature extraction frameworks for dense point cloud maps and their calibration images. This research will explore structured representation rules and lightweight storage mechanisms for dense point cloud maps, providing data support for cross-modal matching of corresponding features between images and point cloud maps during visual localization. High-precision dense point cloud maps with semantic information acquired by laser SLAM can clearly and accurately represent the three-dimensional structural information of real scenes. However, their large data volume, strong discreteness, and large differences in point density are not conducive to data storage and querying. This results in time-consuming and inefficient matching of the same-name features between images and point cloud maps in visual positioning, which cannot meet the needs of real-time visual positioning. It is urgent to solve the problem of lightweight storage of high-precision dense maps, which involves three key technical bottlenecks: (1) The all-day visual positioning requirement poses a major challenge to the feature types stored in the positioning map. Cross-modal matching of visual image features and three-dimensional features of the positioning map is the ultimate goal of visual positioning, but it is extremely challenging. On the contrary, if the all-day (time-segmented) image features of the application scene can be stored in the positioning map to assist visual positioning, it is a feasible solution, but the consistency problem of all-day, multi-view image feature extraction needs to be solved. (2) The aforementioned image features are the bridge between visual images and point cloud maps. Studying the synchronous extraction and lightweight expression rules of point cloud maps and their calibrated image features is another key scientific problem to achieve lightweight map storage. (3) Clarifying the heterogeneous characteristics and intrinsic relationship between the aforementioned point cloud and calibration image, and constructing a storage mechanism for multi-granularity, hierarchical map elements, is the third key unsolvable problem in the lightweight storage of dense maps. Summary of the Invention

[0003] The technical problem this invention aims to solve is as follows: Addressing the aforementioned problems in existing technologies, particularly the high computational load and large storage space requirements of existing indoor enclosed scene representation methods, this invention provides a method and system for constructing a vectorized feature set of an indoor scene point cloud model. The method is simple to implement and can achieve a complete representation of multi-dimensional scene data based on point cloud, visible light, and depth map data. By employing a graph structure for data vectorization and fusion, the computational load can be effectively reduced, enabling efficient and low-consumption construction of a vectorized feature set for an indoor scene point cloud model.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0005] A method for constructing a vectorized feature set for an indoor scene point cloud model, comprising:

[0006] S101 collects point cloud data, visible light images from different directions, and depth maps for indoor scenes.

[0007] S102, performs target object segmentation on point cloud data based on semantics, and performs image segmentation on visible light images and depth maps based on edges;

[0008] S103, based on the segmentation results of target object segmentation and image segmentation, extract the three-dimensional features of the closed bounding region from the point cloud, visible light image and depth map respectively;

[0009] S104 will fuse the 3D features extracted from point cloud, visible light image and depth map;

[0010] S105, a vectorized representation of spatial 3D visual data of an indoor scene generated based on fused 3D elements.

[0011] Optionally, step S102, which involves semantic-based target object segmentation of the point cloud, includes:

[0012] S201, remove noisy point clouds from the collected point cloud data P of size N×3, where N is the number of points and 3 represents the xyz three-dimensional coordinates represented by each point;

[0013] S202, perform semantic segmentation on the point cloud data P after removing noise, and classify the semantic affiliation of each point to obtain the probability of each point belonging to each of the K classes. The probability ranges from 0 to 1. Take the highest probability of each point as the classification result. The output data size is (NM)×1. Point clouds belonging to the same class are considered as one segmentation result, where N is the number of points in each dimension of point cloud P, and M is the number of noise-removed point clouds in each dimension.

[0014] Optionally, step S201, which involves removing noise from the acquired point cloud data P of size N×3, includes: extracting features from the point cloud data of size N×3 to obtain features F of size N×C, where C is a specified feature dimension; calculating the mean feature vector 1×C in the feature F of size N×C; calculating the Euclidean distance of all points in the point cloud data of size N×3 relative to the mean feature vector 1×C; and discarding points whose Euclidean distance exceeds a preset threshold distance δ to obtain point cloud data P of size (NM)×3 after removing noise, where M is the number of noise points removed.

[0015] Optionally, step S102, which involves edge-based image segmentation of the visible light image and depth map, includes:

[0016] S301, Perform resolution measurement on visible light image I to obtain the size H of visible light image I. I ×W I ; Perform resolution measurement on depth map D to obtain the size H of depth map D. D ×W D ;

[0017] S302, scale and align the visible light image I and the depth map D to make them the same size, perform semantic classification on each pixel in the visible light image I, and obtain the probability of each pixel belonging to each of the K classes. The probability ranges from 0 to 1. Take the highest probability of each point as the classification result to achieve image segmentation of the visible light image I.

[0018] Optionally, step S102, which involves performing image segmentation on the depth map based on edges, includes: classifying the distribution of depth values ​​in the depth map D, with the classification based on the numerical value corresponding to the depth map at that point, and uniformly dividing the maximum depth range corresponding to the depth map into k intervals, where the pixels in each interval belong to the same category to achieve image segmentation of the depth map D.

[0019] Optionally, step S103 includes: for the segmentation result of the target object segmentation, vectorizing the points belonging to the same class, and representing the original point cloud data of size N×3 into a segmented point cloud of size k×M×3 according to the classification result, where k is the classification category, M is the number of points in each category, and 3 represents the xyz three-dimensional coordinates represented by each point; for the segmentation result of the visible light image I, vectorizing the visible light pixels belonging to the same class, representing them as a segmented visible light map of size k×i×j×3, where k is the classification category, i×j is the size of each classification region, and 3 represents the RGB color value corresponding to each pixel; for the segmentation result of the depth map D, vectorizing the depth light pixels belonging to the same class, representing them as a segmented depth map of size k×i×j×1, where k is the classification category, i×j is the size of each classification region, and 3 represents the one-dimensional depth value corresponding to each pixel.

[0020] Optionally, step S104 includes: fusing a segmented point cloud of size k×M×3, a segmented visible light map of size k×i×j×3, and a segmented depth map of size k×i×j×1 to obtain a multi-source segmentation feature set of size k×(i×j+M)×7, where k is the semantic category of the object in the scene.

[0021] Optionally, step S105 includes: averaging the dimensions of points in each category of the multi-source segmentation feature set with size k×(i×j+M)×7 to obtain points with dimension 4, resulting in a fused multi-source segmentation feature set with size k×(i×j+M)×4, where 4 indicates that the point has 4 dimensions. The 4 dimensions refer to x, y, z, d or r, g, b, d, where x, y, z are the three dimensions of the three-dimensional coordinates, r, g, b are the dimensions of the red, green, and blue color channels, and d is the depth dimension.

[0022] Furthermore, the present invention also provides a vectorized feature set construction system for an indoor scene point cloud model, including a microprocessor and a memory interconnected thereto, wherein the microprocessor is programmed or configured to execute the vectorized feature set construction method for the indoor scene point cloud model.

[0023] Furthermore, the present invention also provides a computer-readable storage medium storing a computer program, the computer program being programmed or configured by a microprocessor to execute a method for constructing a vectorized feature set of the indoor scene point cloud model.

[0024] Compared with existing technologies, the present invention has the following advantages: The method of the present invention includes collecting point cloud data, visible light images from different directions, and depth maps for indoor scenes; performing target object segmentation on the point cloud data based on semantics, and performing image segmentation on the visible light images and depth maps based on edges; extracting three-dimensional elements of enclosed regions from the point cloud, visible light images, and depth maps based on the segmentation results, and then fusing them to generate a spatial three-dimensional visual data vectorized representation of the indoor scene. The implementation method of the present invention is simple and can realize the complete expression of multi-dimensional scene data based on point cloud, visible light, and depth map data. Due to the use of data vectorization representation and fusion graph structure, the computational load of the method can be effectively reduced, and the vectorized element set construction of the indoor scene point cloud model can be realized efficiently and with low consumption. Attached Figure Description

[0025] Figure 1 This is a schematic diagram of the basic process of the method in an embodiment of the present invention. Detailed Implementation

[0026] like Figure 1 As shown, the method for constructing the vectorized feature set of the indoor scene point cloud model in this embodiment includes:

[0027] S101 collects point cloud data, visible light images from different directions, and depth maps for indoor scenes.

[0028] S102, performs target object segmentation on point cloud data based on semantics, and performs image segmentation on visible light images and depth maps based on edges;

[0029] S103, based on the segmentation results of target object segmentation and image segmentation, extract the three-dimensional features of the closed bounding region from the point cloud, visible light image and depth map respectively;

[0030] S104 will fuse the 3D features extracted from point cloud, visible light image and depth map;

[0031] S105, a vectorized representation of spatial 3D visual data of an indoor scene generated based on fused 3D elements.

[0032] In this embodiment, step S102, which involves semantic-based target object segmentation of the point cloud, includes:

[0033] S201, remove noisy point clouds from the collected point cloud data P of size N×3, where N is the number of points and 3 represents the xyz three-dimensional coordinates represented by each point;

[0034] S202, perform semantic segmentation on the point cloud data P after removing noise, and classify the semantic affiliation of each point to obtain the probability of each point belonging to each of the K classes. The probability ranges from 0 to 1. Take the highest probability of each point as the classification result. The output data size is (NM)×1. Point clouds belonging to the same class are considered as one segmentation result, where N is the number of points in each dimension of point cloud P, and M is the number of noise-removed point clouds in each dimension.

[0035] It should be noted that semantic segmentation of the point cloud data P after removing noise can be performed using the required semantic segmentation method. For example, the semantic segmentation method used in this embodiment can be found in: Qi CR, Su H, Mo K, et al. Pointnet: Deep learning on point sets for 3d classification and segmentation[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2017:652-660.

[0036] In this embodiment, step S201, which involves removing noise from the collected point cloud data P of size N×3, includes: extracting features from the point cloud data of size N×3 to obtain features F of size N×C, where C is the specified feature dimension; calculating the mean feature vector 1×C in the feature F of size N×C; calculating the Euclidean distance of all points in the point cloud data of size N×3 relative to the mean feature vector 1×C; and discarding points whose Euclidean distance exceeds a preset threshold distance δ to obtain point cloud data P of size (NM)×3 after removing noise, where M is the number of noise points removed.

[0037] In this embodiment, step S102, which involves image segmentation based on edges of the visible light image and depth map, includes:

[0038] S301, Perform resolution measurement on visible light image I to obtain the size H of visible light image I. I ×W I ; Perform resolution measurement on depth map D to obtain the size H of depth map D. D ×W D ;

[0039] S302, scale and align the visible light image I and the depth map D to make them the same size, perform semantic classification on each pixel in the visible light image I, and obtain the probability of each pixel belonging to each of the K classes. The probability ranges from 0 to 1. Take the highest probability of each point as the classification result to achieve image segmentation of the visible light image I.

[0040] It should be noted that the semantic attribution classification of each pixel in the visible light image I can be performed using the required semantic attribution classification method as needed. For example, the semantic attribution classification method used in this embodiment can be found in: He K, Zhang X, Ren S, et al. Deep residual learning for image recognition[C] / / Proceedings of the IEEE conference on computer vision and pattern recognition.2016:770-778.

[0041] In this embodiment, step S102, which involves performing image segmentation on the depth map based on edges, includes: classifying the distribution of depth values ​​in the depth map D, with the classification based on the numerical value corresponding to the depth map at that point, and uniformly dividing the maximum depth range corresponding to the depth map into k intervals, where the pixels in each interval belong to the same category to achieve image segmentation of the depth map D.

[0042] In this embodiment, step S103 includes: for the segmentation result of the target object segmentation, vectorizing the points belonging to the same category, and representing the original point cloud data of size N×3 into a segmented point cloud of size k×M×3 according to the classification result, where k is the classification category, M is the number of points in each category, and 3 represents the xyz three-dimensional coordinates represented by each point; for the segmentation result of the visible light image I, vectorizing the visible light pixels belonging to the same category, representing them as a segmented visible light map of size k×i×j×3, where k is the classification category, i×j is the size of each classification region, and 3 represents the RGB color value corresponding to each pixel; for the segmentation result of the depth map D, vectorizing the depth light pixels belonging to the same category, representing them as a segmented depth map of size k×i×j×1, where k is the classification category, i×j is the size of each classification region, and 3 represents the one-dimensional depth value corresponding to each pixel.

[0043] In this embodiment, step S104 includes: fusing the segmented point cloud in vector form with size k×M×3, the segmented visible light map in vector form with size k×i×j×3, and the segmented depth map in vector form with size k×i×j×1 to obtain a multi-source segmentation feature set with size k×(i×j+M)×7, where k is the semantic category of the object in the scene.

[0044] In this embodiment, step S105 includes: averaging the dimensions of points in each category of the multi-source segmentation feature set with size k×(i×j+M)×7, ultimately obtaining points with dimension 4, resulting in a fused multi-source segmentation feature set with size k×(i×j+M)×4. Here, 4 indicates that the point possesses four dimensions, which are x, y, z, d or r, g, b, d. x, y, and z are the three dimensions of the three-dimensional coordinates, r, g, and b are the dimensions of the red, green, and blue color channels, and d is the depth dimension. After the above processing, the original point cloud with size N×3 and the point cloud with size H... I ×W I A visible light image of size H × 3. D ×W D The three types of data with a value of ×1 are represented in a lightweight manner as k×(i×j+M)×4, which greatly compresses the total data volume to achieve a lightweight representation.

[0045] In summary, considering the high computational cost and large storage space required for existing indoor enclosed scene representation methods, this embodiment provides a simple implementation method for constructing a vectorized feature set of an indoor scene point cloud model. This method can achieve a complete representation of multi-dimensional scene data based on point cloud, visible light, and depth map data. By adopting a graph structure form that combines data vectorization and fusion, the computational cost can be effectively reduced, enabling efficient and low-cost construction of a vectorized feature set for an indoor scene color point cloud model.

[0046] Furthermore, this embodiment also provides a system for constructing a vectorized feature set of an indoor scene point cloud model, including a microprocessor and a memory interconnected thereon. The microprocessor is programmed or configured to execute a method for constructing a vectorized feature set of the indoor scene point cloud model. Additionally, this embodiment also provides a computer-readable storage medium storing a computer program for being programmed or configured by the microprocessor to execute the method for constructing a vectorized feature set of the indoor scene point cloud model.

[0047] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-readable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0048] The above description is merely a preferred embodiment of the present invention. The scope of protection of the present invention is not limited to the above embodiments. All technical solutions falling within the scope of the present invention's concept are within the scope of protection of the present invention. It should be noted that for those skilled in the art, any improvements and modifications made without departing from the principles of the present invention should also be considered within the scope of protection of the present invention.

Claims

1. A method for constructing a vectorized feature set of an indoor scene point cloud model, characterized in that, include: S101 collects point cloud data, visible light images from different directions, and depth maps for indoor scenes. S102, performs target object segmentation on point cloud data based on semantics, and performs image segmentation on visible light images and depth maps based on edges; S103, based on the segmentation results of target object segmentation and image segmentation, extract the three-dimensional features of the closed bounding region from the point cloud, visible light image and depth map respectively; S104 will fuse the 3D features extracted from point cloud, visible light image and depth map; S105, Based on the fused 3D elements, generate a spatial 3D visual data vector representation of the indoor scene, including: for a size of The dimensions of points in each category of the multi-source segmentation feature set are averaged to obtain points with dimension 4, resulting in a size of [missing value]. The fusion of multi-source segmentation feature sets, where k represents the classification category. The size of each category region, The number of points in each category, 4 indicates that the point has 4 dimensions. The 4 dimensions refer to x, y, z, d or r, g, b, d. x, y, z are the three dimensions of the three-dimensional coordinates, r, g, b are the dimensions of the three color channels (red, green, and blue), and d is the depth dimension.

2. The method for constructing a vectorized feature set of an indoor scene point cloud model according to claim 1, characterized in that, Step S102, which involves semantic-based target object segmentation of the point cloud, includes: S201, for the acquired dimensions are Point cloud data Remove noisy point clouds, among which The number of points is 3, where 3 represents the xyz three-dimensional coordinates represented by each point. S202, for point cloud data after removing noise. Semantic segmentation is performed, and each point is semantically classified to obtain the probability that each point belongs to each of the K classes. The probability ranges from 0 to 1. The highest probability for each point is taken as the classification result. The output data size is [size missing]. Point clouds belonging to the same class are considered as one segmentation result, where For point clouds M represents the number of points in each dimension, where M is the number of point clouds with noise removed in each dimension.

3. The method for constructing a vectorized feature set of an indoor scene point cloud model according to claim 2, characterized in that, In step S201, the acquired dimensions are... Point cloud data The noise removal process for point clouds includes: extracting features from point cloud data of size N×3 to obtain feature F of size N×C, where C is a specified feature dimension; calculating the mean feature vector 1×C in feature F of size N×C; calculating the Euclidean distance of all points in the point cloud data of size N×3 relative to the mean feature vector 1×C; and discarding points whose Euclidean distance exceeds a preset threshold distance δ to obtain noise-removed point cloud data of size (NM)×3. , where M is the number of noisy point clouds removed.

4. The method for constructing a vectorized feature set of an indoor scene point cloud model according to claim 2, characterized in that, Step S102, which involves edge-based image segmentation of the visible light image and depth map, includes: S301, for visible light images Resolution measurements were performed to obtain visible light images. Size ; Perform resolution measurement on depth map D to obtain the size of depth map D. ; S302, visible light image Scaling and aligning with depth map D to make them the same size for visible light images Each pixel in the image is semantically classified to obtain the probability of each pixel belonging to each of the K classes. The probability ranges from 0 to 1. The highest probability for each pixel is taken as the classification result to achieve the classification of visible light images. Image segmentation.

5. The method for constructing a vectorized feature set of an indoor scene point cloud model according to claim 4, characterized in that, Step S102, which involves edge-based image segmentation of the depth map, includes: classifying the depth value distribution in the depth map D based on the numerical value corresponding to the depth map at that point, and uniformly dividing the maximum depth range corresponding to the depth map into... k The image segmentation of depth map D is achieved by dividing the pixels in each interval into three ranges, with each range belonging to the same category.

6. The method for constructing a vectorized feature set of an indoor scene point cloud model according to claim 5, characterized in that, Step S103 includes: For the segmentation results of the target object segmentation, vectorizing the points belonging to the same class, representing the original point cloud data of size N×3 into a segmented point cloud of size k×M×3 according to the classification results, where k is the classification category, M is the number of points in each category, and 3 represents the xyz three-dimensional coordinates represented by each point; for visible light images... The segmentation results are then used to vectorize visible light pixels belonging to the same category, represented as pixels of size [size missing]. The visible light map is segmented in vector form, where 3 represents the RGB color value corresponding to each pixel. Based on the segmentation result of the depth map D, depth light pixels belonging to the same category are vectorized, represented as a vector of size [size missing]. The segmented depth map is in vector form, where 1 represents the one-dimensional depth value corresponding to each pixel.

7. The method for constructing a vectorized feature set of an indoor scene point cloud model according to claim 6, characterized in that, Step S104 includes: dividing the point cloud into a vector form with dimensions k×M×3 and a point cloud with dimensions k×M×3. The vector form of the segmented visible light map and the size of The vector form of the segmented depth map is fused to obtain a size of The multi-source segmentation feature set, where k is the classification category.

8. A system for constructing a vectorized feature set of an indoor scene point cloud model, comprising a microprocessor and a memory interconnected, characterized in that, The microprocessor is programmed or configured to execute the method for constructing a vectorized feature set of an indoor scene point cloud model according to any one of claims 1 to 7.

9. A computer-readable storage medium storing a computer program, characterized in that, The computer program is used to be programmed or configured by a microprocessor to execute the method for constructing a vectorized feature set of an indoor scene point cloud model according to any one of claims 1 to 7.

Images