Visual feature and geometric feature fusion method, registration method and related product

Abstract

The embodiment of the invention provides a visual feature and geometric feature fusion method, a registration method and a related product, and the fusion method comprises the steps: obtaining a visual feature point set of a two-dimensional image, and obtaining a geometric feature point set of a three-dimensional point cloud corresponding to the two-dimensional image, carrying out feature point supplementation on an area with relatively sparse visual feature points in the two-dimensional image to obtain an enhanced feature point set; determining feature points with mutually corresponding pixel point coordinates in the enhanced feature point set and the geometric feature point set as mapping feature points; performing visual feature coding on the mapping feature points in the two-dimensional image; determining a visual weight and a geometric weight corresponding to each mapping feature point based on the visual coding value after normalization processing; and based on the visual weight, the visual feature value, the geometric weight and the geometric feature value, determining a fusion feature value of each mapping feature point. According to the method, fusion feature points with richer feature expression ability can be provided, so that the accuracy and stability of point cloud registration are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional point clouds, and in particular to a method for fusing visual features with geometric features, a point cloud registration method, an electronic device, and a computer-readable storage medium. Background Art

[0002] Point cloud registration plays a crucial role in 3D reconstruction, precisely aligning point cloud data collected from different viewpoints or at different times to generate a complete 3D model. Point cloud registration is widely used in fields such as industrial inspection, reverse engineering, cultural heritage preservation, and architectural modeling, providing high-quality 3D data for subsequent data analysis, visualization, and decision-making. Accurate point cloud registration ensures spatial consistency, thereby improving the accuracy and completeness of 3D reconstructions and providing a reliable foundation for understanding 3D environments in a variety of application scenarios.

[0003] Point cloud registration refers to the process of aligning two or more point cloud data acquired from different perspectives, at different times, or using different devices into the same coordinate system. Specifically, point cloud registration requires finding a transformation matrix that aligns one point cloud with another. Existing point cloud registration methods often rely solely on two-dimensional visual features or three-dimensional geometric features. When dealing with weakly textured areas, the expressive power of a single feature is insufficient to provide sufficient registration basis, leading to point cloud matching failures or mismatches. Summary of the Invention

[0004] Based on the above situation, the main purpose of the present invention is to provide a method for fusing visual features and geometric features, a point cloud registration method and related technical solutions, which can provide fused feature points with richer feature expression capabilities as matching objects in the point cloud registration task, so that the point cloud registration in weak texture scenes can be more accurate and stable.

[0005] To achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A first aspect of an embodiment of the present invention provides a method for fusing visual features with geometric features, including:

[0007] Acquire a set of visual feature points of a two-dimensional image, the set of visual feature points including visual feature points and their corresponding visual feature values; acquire a set of geometric feature points of a three-dimensional point cloud corresponding to the two-dimensional image, the set of geometric feature points including geometric feature points and their corresponding geometric feature values;

[0008] Supplementing feature points in an area of ​​the two-dimensional image where visual feature points are relatively sparse to obtain a set of enhanced feature points, wherein the set of enhanced feature points includes visual feature points and supplemented feature points;

[0009] Determine feature points whose pixel coordinates in the enhanced feature point set and the geometric feature point set correspond to each other as mapping feature points;

[0010] Performing visual feature encoding on the mapped feature points in the two-dimensional image to obtain a visual encoding value corresponding to each of the mapped feature points; determining a visual weight and a geometric weight corresponding to each of the mapped feature points based on the normalized visual encoding value, wherein the visual encoding value of a supplementary feature point among the mapped feature points is zero, and the sum of the visual weight and the geometric weight is a unit matrix; and

[0011] The fusion feature value of each mapping feature point is determined based on the visual weight, the visual feature value, the geometric weight, and the geometric feature value corresponding to each mapping feature point.

[0012] Preferably, the feature point supplementation for an area in the two-dimensional image where visual feature points are relatively sparse includes: obtaining visual feature points corresponding to a foreground area in the two-dimensional image; generating a local neighborhood with each of the visual feature points as the center, the local neighborhood being a two-dimensional Gaussian kernel whose radiation value decays sequentially from the center to the surrounding edges; accumulating the radiation values ​​formed by each of the local neighborhoods for any pixel point in the foreground area to obtain the cumulative radiation values ​​corresponding to each pixel point in the foreground area, and determining the mean of the cumulative radiation values ​​in the foreground area as a recognition threshold; traversing each pixel point in the foreground area based on the recognition threshold, and screening out N pixel points whose cumulative radiation values ​​are less than the recognition threshold; and randomly selecting M pixel points from the N pixel points as supplementary feature points based on a preset feature point density, where M is an integer less than N.

[0013] Preferably, the feature point supplementation for an area in the two-dimensional image where visual feature points are relatively sparse includes: gridding the two-dimensional image to obtain a plurality of grid areas; determining a foreground area and a background area of ​​the two-dimensional image, and determining an edge grid area and an internal grid area among the plurality of grid areas, wherein the edge grid area is a grid area having both foreground pixels and background pixels, and the internal grid area is a grid area containing only foreground pixels; respectively determining the feature point density of the edge grid area and the internal grid area, wherein the feature point density of the internal grid area is a preset baseline feature point density, and the feature point density of the edge grid area is the product of the foreground proportion and the baseline feature point density, and the foreground proportion is the foreground pixel density within the edge grid area. The ratio of the number of scene pixels to the number of all pixels in the edge grid area; a local neighborhood is generated with each of the visual feature points in the foreground area as the center, and the local neighborhood is a two-dimensional Gaussian kernel whose radiation value decays from the center to the surrounding edges; the radiation value formed by each of the local neighborhoods for any pixel point in the foreground area is accumulated to obtain the cumulative radiation value corresponding to each pixel point in the foreground area, and the mean of the cumulative radiation values ​​in the foreground area is determined as the recognition threshold; based on the recognition threshold, it is determined whether each of the edge grid areas and each of the internal grid areas needs to be supplemented with feature points; and based on the feature point density corresponding to each of the edge grid areas and the internal grid areas, the grid areas that need to be supplemented with feature points are randomly inserted with supplementary feature points.

[0014] Preferably, the visual feature encoding of the mapped feature points in the two-dimensional image includes: performing the following processing on each visual feature point in the visual feature point set to obtain a visual encoding value of each visual feature point: obtaining 8 neighborhood pixels of the current visual feature point; comparing the grayscale values ​​of the current visual feature point with each of the neighborhood pixels, marking the neighborhood pixels with grayscale values ​​greater than or equal to the current visual feature point as 1, and marking the neighborhood pixels with grayscale values ​​less than the current visual feature point as 0; arranging the mark values ​​of the 8 neighborhood pixels in sequence into a group of binary numbers; in the binary number, the situation where any two adjacent bits have different values ​​is regarded as a jump, and if the number of jumps is less than or equal to 2, setting the visual encoding value according to the number marked as 1 in the binary number; if the number of jumps is greater than 2, setting the visual encoding value to a preset value.

[0015] Preferably, the visual coding value after normalization is used to determine the visual weight and geometric weight corresponding to each of the mapping feature points, including: obtaining the first visual coding values ​​of m mapping feature points in the first point cloud and the second visual coding values ​​of n mapping feature points in the second point cloud respectively; and performing the following processing on any mapping feature point i among the m mapping feature points: summing the first visual coding value of the current mapping feature point i with the n second visual coding values ​​and taking the average to obtain a 1*n weight matrix; normalizing the weight matrix to obtain the visual weight; subtracting the visual weight from the 1*n unit matrix to obtain the geometric weight, and the visual weight and the geometric weight are both 1*n normalized matrices.

[0016] Preferably, the normalizing the weight matrix includes: normalizing any weight value in the 1*n weight matrix based on the following relationship:

[0017]

[0018] Where W 2d (i, j) represents the weight value of the current normalization process, j is any one of the n second visual coding values, W min is the one with the smallest visual encoding value among the n weight values, W max is the one with the largest visual encoding value among the n weight values, is the normalized visual weight value.

[0019] Preferably, determining the fused feature value of each mapping feature point based on the visual weight, the visual feature value, the geometric weight, and the geometric feature value corresponding to each mapping feature point includes: determining the fused feature value of any mapping feature point based on the following relationship:

[0020]

[0021] Where D f is the fusion feature value, is the visual weight, D 2d is the visual feature value, is the geometric weight, D 3d is the geometric characteristic value.

[0022] A second aspect of an embodiment of the present invention provides a point cloud registration method based on fusion features, comprising:

[0023] Determine, using the fusion method described in the first aspect, a first fused feature value corresponding to a first mapping feature point in the first point cloud and a second fused feature value corresponding to a second mapping feature point in the second point cloud;

[0024] Determining the nearest neighbor of each of the first mapping feature points by using the following method: selecting any unprocessed first mapping feature point in the first point cloud, determining a first feature similarity between the first mapping feature point and each of the second mapping feature points, and selecting the second mapping feature point corresponding to the second mapping feature point with the highest first feature similarity as the nearest neighbor of the first mapping feature point;

[0025] determining a set of feature point pairs of the first point cloud and the second point cloud based on the nearest neighbors of each of the first mapped feature points;

[0026] constructing a compatibility graph based on the set of feature point pairs, searching for a maximal clique in the compatibility graph to obtain a maximal clique set; and

[0027] Perform singular value decomposition on each of the maximal clusters in the maximal cluster set to obtain a posture hypothesis corresponding to each of the maximal clusters, and select the optimal one among the posture hypotheses as the transformation matrix for point cloud registration between the first point cloud and the second point cloud.

[0028] Preferably, before determining a set of feature point pairs of the first point cloud and the second point cloud based on the nearest neighbors of each of the first mapped feature points, the method further includes:

[0029] Determining the nearest neighbor of each second mapping feature point by using the following method: selecting any unprocessed second mapping feature point in the second point cloud, determining the second feature similarity between the second mapping feature point and each first mapping feature point, and using the first mapping feature point corresponding to the one with the highest second feature similarity as the nearest neighbor of the second mapping feature point;

[0030] The determining of a set of feature point pairs of the first point cloud and the second point cloud based on the nearest neighbors of each of the first mapping feature points includes: selecting mapping feature points that are each other's nearest neighbors in the first point cloud and the second point cloud as feature point pairs, and determining a set of feature point pairs of the first point cloud and the second point cloud.

[0031] Preferably, the method further comprises: determining the feature similarity between the first mapping feature point and the second mapping feature point using the following relationship:

[0032]

[0033] Where S(i,j) represents the feature similarity of feature point pair (i,j), i represents any first mapping feature point, j represents any second mapping feature point, Indicates the jth weight value of the current first mapping feature point i in the 1*n visual weight, Indicates the jth weight value in the 1*n geometric weight, S 2d (i, j) represents the visual feature similarity between the first point cloud and the second point cloud, S 3d (i, j) represents the geometric feature similarity between the first point cloud and the second point cloud, D 2d (i) represents the visual feature value of the first mapping feature point i, D 2d (j) represents the visual feature value of the second mapping feature point j, D 3d (i) represents the geometric eigenvalue of the first mapping feature point i, D 3d (j) represents the geometric eigenvalue of the second mapping feature point j.

[0034] The third aspect of the embodiment of the present invention provides a computer program stored thereon, and when the computer program is executed by a processor, it implements the method for fusing visual features and geometric features of the first aspect, or a point cloud registration method based on fusion features provided by the second aspect.

[0035] An electronic device provided in the fourth aspect of an embodiment of the present invention includes a storage medium storing a computer program. When the computer program is executed by a processor, it implements the method for fusing visual features and geometric features of the first aspect above, or a point cloud registration method based on fused features provided in the second aspect.

[0036] In an embodiment of the present invention, on the one hand, the fusion weights of the mapping feature points are set based on the visual feature encoding, and the mapping feature points in the feature point sets of the two-dimensional image and the three-dimensional point cloud are fused, which makes up for the lack of single feature expression ability in the point cloud registration process, and helps to improve the stability and accuracy of point cloud registration in weak texture scenes. On the other hand, during the fusion process, due to factors such as insufficient feature points in the weak texture area, hardware errors of different sensors, occlusion, etc., not all feature points in the three-dimensional point cloud can find corresponding mapping feature points in the visual feature point set. The embodiment of the present invention supplements the feature points in the area where the visual feature points in the two-dimensional image are relatively sparse, and the number of mappable feature points in the obtained enhanced feature point set is increased, so that more mapping feature points can be matched in the subsequent point cloud registration process, thereby improving the accuracy of point cloud registration in weak texture scenes.

[0037] Other beneficial effects of the present invention will be explained through the introduction of specific technical features and technical solutions in the specific implementation methods. Those skilled in the art should be able to understand the beneficial technical effects brought about by the introduction of these technical features and technical solutions. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The preferred embodiments of the present invention will be described below with reference to the accompanying drawings.

[0039] Figure 1 A flow chart showing a method for fusing visual features and geometric features according to an embodiment of the present invention is shown;

[0040] Figure 2 A flow chart showing a method for supplementing feature points according to an embodiment of the present invention is shown;

[0041] Figure 3 A schematic diagram showing a characteristic point density radiation field according to an embodiment of the present invention is shown;

[0042] Figure 4 A schematic diagram showing visual feature encoding according to an embodiment of the present invention is shown;

[0043] Figure 5 Another flow chart of the feature point supplementation method according to an embodiment of the present invention is shown;

[0044] Figure 6 A schematic diagram showing grid division in an embodiment of the present invention is shown;

[0045] Figure 7 A flow chart of a point cloud registration method according to an embodiment of the present invention is shown;

[0046] Figure 8 A schematic diagram illustrating construction of a maximal clique in an embodiment of the present invention is shown;

[0047] Figure 9 A schematic diagram of the structure of a point cloud registration method according to an embodiment of the present invention is shown;

[0048] Figure 10 A schematic diagram of an electronic device according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0049] The present invention is described below based on the following embodiments, but the present invention is not limited to these embodiments. In the following detailed description of the present invention, some specific details are described in detail. In order to avoid obscuring the essence of the present invention, well-known methods, processes, procedures, and components are not described in detail.

[0050] Furthermore, persons of ordinary skill in the art will appreciate that the figures provided herein are for illustration purposes only and are not necessarily drawn to scale.

[0051] Unless the context clearly requires otherwise, throughout the specification and claims, the words "include," "comprising," and similar words should be construed in an inclusive sense rather than an exclusive or exhaustive sense; that is, in the sense of "including but not limited to."

[0052] In the description of the present invention, it should be understood that the terms "first", "second", etc. are used for descriptive purposes only and should not be understood to indicate or imply relative importance. In addition, in the description of the present invention, unless otherwise specified, "plurality" means two or more.

[0053] In the point cloud registration task, the uniform distribution and spatial coverage of feature points are crucial to the registration accuracy. However, in actual scenes, areas such as flat object surfaces, large areas without textures, or weak geometric undulations often have problems with sparse texture information and / or single geometric features. Existing feature extraction methods rely on significant texture structures, resulting in sparse feature points extracted in these weak texture areas, and even feature extraction failures. In addition, hardware factors such as sensor calibration errors and thermal drift further aggravate the difficulty of feature extraction in weak texture areas. This problem of insufficient feature point extraction will lead to insufficient local constraints in the point cloud registration process, affecting the stability and accuracy of point cloud registration. In view of this, an embodiment of the present invention provides a method for fusing visual features and geometric features, which can provide fused feature points with richer feature expression capabilities as matching objects in point cloud registration tasks, making point cloud registration in weak texture scenes more stable and accurate.

[0054] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings.

[0055] See also Figure 1 , Figure 1 A flow chart of a method for fusing visual features with geometric features according to an embodiment of the present invention is shown. The embodiment of the present invention provides a method for fusing visual features with geometric features, which includes steps S1 to S5.

[0056] S1. Obtain a set of visual feature points of a two-dimensional image, the set of visual feature points including visual feature points and their corresponding visual feature values; obtain a set of geometric feature points of a three-dimensional point cloud corresponding to the two-dimensional image, the set of geometric feature points including geometric feature points and their corresponding geometric feature values.

[0057] The two-dimensional image may be a visual image captured by a camera sensor, or a grayscale image or RGB image extracted based on the visual image. The three-dimensional point cloud may be a point cloud generated based on the two-dimensional image (with depth information annotation). The points in the three-dimensional point cloud can be mapped one-to-one to the pixels in the two-dimensional image. However, since some pixels in the two-dimensional image have no depth information (i.e., the coordinate value in the depth direction is zero), not all pixels in the two-dimensional image can be mapped to points in the three-dimensional point cloud.

[0058] The set of visual feature points can be a dataset of visual feature points in a two-dimensional image, or a distribution map of visual feature points in a two-dimensional image, including the pixel coordinates and visual feature values ​​of each visual feature point. Visual feature points can represent information such as shape features (e.g., image outline and local shape), texture features (e.g., repetitive patterns, texture details, and surface structure in local image regions), spatial position features (e.g., relative and absolute position information), and some semantic features (e.g., key parts of an object or scene elements). Shape features focus on the object's shape, boundaries, dimensions, and the spatial relationships between its parts. For example, the shape of a circular object can be described by its radius, center position, and boundary curve; the shape of a rectangular object can be described by its length, width, four corner points, and side parallelism. Texture features focus on the regularity of pixel arrangement in an image, as well as the directionality, periodicity, and roughness of the texture. For example, the texture of a piece of cloth can be described by the spacing, direction, and color variation of its stripes; the texture of a patch of grass can be described by the distribution density and length variation of its blades. The visual feature points of a two-dimensional image can be extracted through a two-dimensional feature extractor, such as SIFT (a scale-invariant feature transform model), SuperPoint (a self-supervised feature point detection and description algorithm based on deep learning), and AlexNet (a model that extracts high-level features of an image through a multi-layer convolutional neural network). Among them, the visual feature points extracted by different two-dimensional feature extractors have different tendencies in dimensions such as shape features, texture features, or semantic features. In practical applications, a suitable two-dimensional feature extractor can be selected according to the type of target image and the requirements of the image processing task.

[0059] Regarding the set of geometric feature points, it can be a data set of geometric feature points in a three-dimensional point cloud. A point in a three-dimensional point cloud represents the data information contained in a position in three-dimensional space, and this data information includes the three-dimensional coordinates of the point (i.e., the coordinate values ​​on the X-axis, Y-axis, and Z-axis), and further, may include information such as color, intensity, and normal. In an embodiment of the present invention, the geometric feature points extracted from the three-dimensional point cloud can characterize the local geometric shapes of the three-dimensional point cloud, such as edges, corners, and planes. Specifically, the geometric feature points in the three-dimensional point cloud can be extracted by a three-dimensional feature extractor, such as FCGF (Fully Convolutional Geometric Features) or ISS (Intrinsic Shape Signatures). Among them, FCGF can not only capture the local geometric shapes in the point cloud, but also capture a wide range of spatial contextual information. The feature points extracted by FCGF not only contain local information, but also can reflect the global structure of the point cloud.

[0060] S2. Supplement feature points in an area of ​​the two-dimensional image where visual feature points are relatively sparse, to obtain an enhanced feature point set, where the enhanced feature point set includes visual feature points and supplemented feature points.

[0061] Among them, since the supplementary feature points do not belong to the visual feature points in the two-dimensional image, that is, the supplementary feature points cannot represent the shape features and texture features of the two-dimensional image, therefore, in order not to interfere with the original visual features in the two-dimensional image, the visual feature value of the supplementary feature point is set to zero, and the subsequent visual coding value in the LBP coding is also set to zero.

[0062] Regarding the area where the visual feature points are relatively sparse, the embodiment of the present invention is defined as the area where the geometric features are sparse or the texture features are weak in the two-dimensional image. Specifically, "sparse geometric features" and "weak texture features" can be visually manifested as a flat object surface, a large area without texture or weak geometric undulations, and specifically manifested as insufficient visual feature points in terms of data features. It should be noted that the area where the "geometric features are sparse" or "weak texture features" in the embodiment of the present invention can be relative to the area with more other feature points in the same two-dimensional image, that is, the area where the visual feature points are relatively sparse can be determined based on the number of visual feature points in a certain area and the proportion of all visual feature points in the current two-dimensional image (wherein the threshold for demarcating relatively sparse areas can be pre-set based on prior data of different image types). In addition, the above-mentioned "certain area" can be a fixed area demarcated in the two-dimensional image, or it can be an area formed based on the distribution of visual feature points of the current two-dimensional image, depending on actual needs and is not limited here.

[0063] For example, the embodiment of the present invention proposes a method for supplementing feature points based on the feature point density radiation field, which is used to supplement feature points in an area where visual feature points are relatively sparse in a two-dimensional image. Figure 2 , Figure 2 A flow chart of a feature point supplementation method according to an embodiment of the present invention is shown, which includes steps S21 to S25.

[0064] S21. Obtain visual feature points corresponding to the foreground area in the two-dimensional image.

[0065] For example, the grayscale value of each pixel in the two-dimensional image can be obtained, and an initial threshold is determined based on prior data. The pixels with grayscale values ​​greater than the initial threshold are regarded as the foreground, and the pixels with grayscale values ​​less than the initial threshold are regarded as the background. The grayscale values ​​of all foreground pixels are summed and averaged. Sum the grayscale values ​​of all background pixels and calculate the average Then determine the reference threshold T for dividing the foreground area and background area according to the following relationship: d , where the weight β is between 0.6 and 0.8.

[0066]

[0067] After determining the reference threshold, pixels in the two-dimensional image with grayscale values ​​less than the reference threshold are determined as the background area, and pixels in the two-dimensional image with grayscale values ​​greater than or equal to the reference threshold are determined as the foreground area. In an embodiment of the present invention, an initial threshold can be determined based on prior data, and this initial threshold can be used to initially delineate the foreground and background. However, in actual applications, since different image types have different overall grayscales, the initial threshold can be updated to the reference threshold based on the overall grayscale value of the current two-dimensional image to more accurately delineate the foreground and background areas.

[0068] It should be noted that, in practical applications, there are many ways to determine the foreground area of ​​an image. The above method of defining the foreground area based on the reference threshold is only an example and should not be understood as the only implementation method for determining the foreground area.

[0069] S22. Generate a local neighborhood with each of the visual feature points as the center. The local neighborhood is a two-dimensional Gaussian kernel whose radiation value decays from the center to the surrounding edges.

[0070] For example, each feature point in the visual feature point set is treated as a local Gaussian radiation source, which produces an attenuation effect in its neighboring area. This attenuation process is described by a two-dimensional Gaussian kernel, which is mathematically expressed as:

[0071]

[0072] Where σ = r / 2, r is the neighborhood radius of the feature point, and the two-dimensional Gaussian kernel G(x, y) characterizes the radiant influence of the visual feature point on the surrounding space. The farther away from the feature point, the weaker the influence. In this embodiment of the present invention, the neighborhood radius r can be set to 10 pixels. It is understood that the neighborhood radius r can be adjusted according to actual needs, and the specific value is not limited here.

[0073] For any pixel point (x, y) in the space, it receives the information from the feature point (x i ,y i ) can be expressed as:

[0074] R i (x,y)=G(xx i ,yy i )

[0075] Among them, G(x) is the Gaussian weight, and the above relationship reflects the description of the weighted radiation value of the visual feature point to its neighborhood pixels.

[0076] S23. Accumulate the radiation values ​​generated by each local neighborhood for any pixel point in the foreground area to obtain the cumulative radiation values ​​corresponding to each pixel point in the foreground area, and determine the average of the cumulative radiation values ​​in the foreground area as the recognition threshold.

[0077] For example, see Figure 3 , Figure 3 A schematic diagram of the feature point density radiation field in an embodiment of the present invention is shown. The left image shows the distribution of the viewing angle feature points (red dots in the image) in the two-dimensional image in the visual feature point set, the middle image shows the density radiation field composed of the cumulative radiation values ​​of all pixels, and the right image shows the distribution of the viewing angle feature points (red dots in the image) and supplementary feature points (green dots in the image) in the enhanced feature point set in the two-dimensional image. Specifically, the calculation method of the cumulative radiation value can refer to the following relationship:

[0078]

[0079] Where D(x, y) is the cumulative radiation value at the position (x, y), N is the total number of feature points, R i (x, y) is the contribution of the i-th feature point to the pixel (x, y). The mean of the cumulative radiation values ​​in the foreground area can be used as the recognition threshold, and the pixels with cumulative radiation values ​​less than the mean are used as the area to be randomly inserted with feature points. Figure 3As shown in the figure, the cumulative radiation value of each pixel in the foreground area can be calculated, and the cumulative radiation values ​​of all pixels constitute a density radiation field. The density radiation field not only reflects the local aggregation degree of the visual feature points, but also can quantitatively describe the uniformity of the overall distribution of the visual feature points, providing a quantitative basis for the subsequent identification of relatively sparse areas of visual feature points.

[0080] S24. Traverse each pixel point in the foreground area based on the recognition threshold, and screen out N pixel points whose cumulative radiation values ​​are less than the recognition threshold.

[0081] Specifically, the area where the N pixels are located corresponds to an area where visual feature points are relatively sparse in the two-dimensional image, and N is an integer greater than one.

[0082] S25. Based on a preset feature point density, randomly select M pixels from N pixels as supplementary feature points, where M is an integer less than N.

[0083] Specifically, the feature point density can be a threshold value preset based on prior data. For example, multiple feature point extractions are performed on the same type of plaster statue at different image sizes to determine the lower limit of the number of feature points that can be used for effective point cloud registration in a feature-sparse area at the same image size. The feature point density of such images is set based on the size of the feature-sparse area and the lower limit of the number of feature points.

[0084] For example, assuming that the preset feature point density is 1 / 10 (i.e., one pixel needs to be selected as a feature point from 10 pixels), the specific number of M is determined according to the ratio of 1 / 10, and a random sampling method without replacement is adopted to randomly select M pixels from N pixels as supplementary feature points. The supplementary feature points, together with the visual feature points obtained in step S1, serve as feature points of the enhanced feature point set.

[0085] The method for supplementing feature points based on the feature point density radiation field in the embodiments of the present invention can effectively improve the overall spatial uniformity of visual feature points, avoid excessive concentration of visual feature points in high-texture areas, and simultaneously compensate for the matching difficulties and insufficient geometric constraints caused by sparse feature points. This method is not only applicable to typical texture-missing scenarios, but also adapts to local feature point loss caused by factors such as sensor noise interference and perspective changes, providing a more stable and reliable feature foundation for subsequent point cloud registration.

[0086] S3. Determine feature points whose pixel coordinates in the enhanced feature point set and the geometric feature point set correspond to each other as mapping feature points.

[0087] Specifically, because the pixel coordinates of geometric feature points are three-dimensional, while the feature points within the enhanced feature point set are two-dimensional, the "corresponding pixel coordinates" described in step S3 refer to the consistency of the coordinate values ​​on the X and Y axes. In this embodiment of the present invention, mapping the feature points of the two-dimensional image with the feature points of the three-dimensional point cloud can effectively eliminate interference from some noise points.

[0088] S4. Perform visual feature encoding on the mapping feature points in the two-dimensional image to obtain a visual encoding value corresponding to each mapping feature point; and determine a visual weight and a geometric weight corresponding to each mapping feature point based on the normalized visual encoding value.

[0089] Among them, since the supplementary feature points do not have two-dimensional viewing features, the visual encoding value corresponding to the supplementary feature points in the mapping feature points is zero; and since the sum of the visual weight and the geometric weight is limited to the unit matrix in the embodiment of the present invention (that is, in the matrices corresponding to the visual weight and the geometric weight, the sum of each corresponding position element is one), therefore, in the fusion feature corresponding to the supplementary feature point, the weight value of the geometric weight is one (or the unit matrix), that is, the fusion feature corresponding to the supplementary feature point only considers the geometric feature value of the three-dimensional point cloud.

[0090] For example, for the visual feature encoding of visual feature points, reference can be made to some processing methods of Local Binary Patterns (LBP) encoding. Traditional LBP is a feature extraction method used for texture analysis and image description. Its core idea is to compare each pixel in the image with the pixels in its neighborhood to generate a binary number to describe the texture features of the image. In the embodiment of the present invention, the following processing is performed on each visual feature point in the visual feature point set:

[0091] Obtain the 8 neighboring pixels of the current visual feature point; compare the grayscale values ​​of the current visual feature point with those of each neighboring pixel point, mark the neighboring pixel points with grayscale values ​​greater than or equal to the current visual feature point as 1, and mark the neighboring pixel points with grayscale values ​​less than the current visual feature point as 0; arrange the mark values ​​of the 8 neighboring pixel points in order into a group of binary numbers; in the binary number, the situation where any two adjacent bits have different values ​​is regarded as a jump. If the number of jumps is less than or equal to 2, set the visual coding value according to the number of 1s in the binary number; if the number of jumps is greater than 2, set the visual coding value of the current visual feature point to the preset value.

[0092] For example, see Figure 4 , Figure 4A schematic diagram of visual feature encoding in an embodiment of the present invention is shown. The center of the 3*3 matrix in the figure is the visual feature point currently being encoded. The first 3*3 matrix in the second row of the figure shows the grayscale values ​​of the visual feature point and its neighborhood. The second 3*3 matrix in the second row of the figure shows the label values ​​of the neighboring pixels after grayscale value comparison. The 3*3 matrix in the first row of the figure shows the binary bits from left to right after visual encoding.

[0093] In the embodiment of the present invention, the Uniform LBP coding mode (i.e., uniform local binary mode) is adopted, which stipulates that if the encoded binary number jumps more than twice, it is regarded as an uneven mode. That is, the current visual feature point presents relatively disordered feature information around it, which may be caused by noise or slight grayscale changes. This uneven mode may not have practical distinguishing significance in texture analysis. Therefore, the binary number that jumps more than twice is converted into a fixed preset value (please refer to the table below, which is set to "9" in the embodiment of the present invention). If the encoded binary number jumps twice or less, it is regarded as a uniform pattern, that is, the current visual feature point presents obvious texture features. The continuous "1" or "0" in the binary number can present different texture uniformity, texture directionality and texture periodicity. The embodiment of the present invention sets the visual coding value based on the number marked as 1 in the binary number. Specifically, please refer to the table below. If there is no "1" mark bit in the binary number, the visual coding value is 0. If there is one continuous "1", the visual coding value is 1. If there are two continuous "1", the visual coding value is 2. And so on. If there are eight continuous "1", the visual coding value is 8.

[0094] Visual Coding Table

[0095]

[0096] In step S4, after obtaining the visual coding values ​​of all visual feature points in the visual feature point set, the visual weights of the corresponding visual feature points can be set based on the visual coding values. Specifically:

[0097] In one embodiment, a visual weight can be set based solely on the visual feature points corresponding to the current point cloud. For example, the visual coding value of the current visual feature point is used as the visual weight value before normalization. The maximum visual coding value in the set of visual feature points is determined, and the ratio of the visual coding value of the current visual feature point to the maximum visual coding value is used as the normalized visual weight. Accordingly, the geometric weight is the difference between one and the visual weight.

[0098] In another embodiment, visual weights can be set based on the visual feature points corresponding to the source point cloud (first point cloud) and the target point cloud (second point cloud). For example, the first visual coding values ​​of the m mapping feature points in the first point cloud and the second visual coding values ​​of the n mapping feature points in the second point cloud are obtained respectively; for any mapping feature point i among the m mapping feature points, the following processing is performed: the first visual coding value of the current mapping feature point i is summed with the n second visual coding values ​​and the average is taken to obtain a 1*n weight matrix; the weight matrix is ​​normalized to obtain the visual weight; the visual weight is subtracted from the 1*n unit matrix to obtain the geometric weight, and the visual weight and the geometric weight are both 1*n normalized matrices. Specifically, you can refer to the following relationship:

[0099]

[0100] Where W 2d (i,j) represents the jth weight value of the current mapping feature point i in the 1*n weight matrix, LBP1(i) represents the first visual coding value corresponding to the current first mapping feature point i, and LBP2(i) represents the second visual coding value corresponding to the second mapping feature point j.

[0101] For example, the weight matrix can be normalized by referring to the following relationship:

[0102]

[0103] Where W 2d (i, j) is the weight value of the current normalization process, j is any one of the n second visual coding values, W min is the one with the smallest visual encoding value among the n weight values, W max is the one with the largest visual encoding value among the n weight values, is the normalized visual weight value.

[0104] For example, the geometric weight may be determined by referring to the following relationship:

[0105]

[0106] Where, is the normalized geometric weight value. For a single mapping feature point, “I” represents the 1*n identity matrix; for all m mapping feature points in the first point cloud, “I” represents the m*n identity matrix.

[0107] In the scheme for determining visual weights based on the first point cloud and the second point cloud, for any mapping feature point corresponding to the first point cloud, it is equivalent to constructing an association relationship between the mapping feature point and each mapping feature point in the second point cloud, so as to facilitate a more accurate subsequent estimation of the distance (cosine similarity) from the mapping feature point of the first point cloud to the mapping feature point of the second point cloud.

[0108] S5. Determine the fusion feature value of each mapping feature point based on the visual weight, visual feature value, geometric weight, and geometric feature value corresponding to each mapping feature point.

[0109] For example, the fusion feature value of any mapping feature point can be determined based on the following relationship:

[0110]

[0111] Where D f is the fusion feature value, is the visual weight, D 2d is the visual feature value, is the geometric weight, D 3d is the geometric characteristic value.

[0112] In an embodiment of the present invention, on the one hand, the fusion weights of the mapping feature points are set based on the visual feature encoding, and the mapping feature points in the feature point sets of the two-dimensional image and the three-dimensional point cloud are fused, which makes up for the lack of single feature expression ability in the point cloud registration process, and helps to improve the stability and accuracy of point cloud registration in weak texture scenes. On the other hand, during the fusion process, due to factors such as insufficient feature points in the weak texture area, hardware errors of different sensors, occlusion, etc., not all feature points in the three-dimensional point cloud can find corresponding mapping feature points in the visual feature point set. The embodiment of the present invention supplements the feature points in the area where the visual feature points in the two-dimensional image are relatively sparse, and the number of mappable feature points in the obtained enhanced feature point set is increased, so that more mapping feature points can be matched in the subsequent point cloud registration process, thereby improving the accuracy of point cloud registration in weak texture scenes.

[0113] In step S2, for the method of supplementing feature points in a relatively sparse area of ​​visual feature points, there is another embodiment of the present invention, which can be found in Figure 5 , Figure 5 Another flow chart of the feature point supplementation method in an embodiment of the present invention is shown, which includes steps H21 to H27.

[0114] H21. Divide the two-dimensional image into grids to obtain multiple grid areas.

[0115] For example, see Figure 6 , Figure 6 A schematic diagram of grid division in an embodiment of the present invention is shown. In order to uniformly insert feature points in relatively sparse areas, the two-dimensional image can be first divided into grid areas, and each grid area is numbered so that feature points can be randomly inserted into each grid area independently later.

[0116] It should be noted that the size of a single grid area may be determined according to specific uniformity requirements and computational efficiency requirements, and is not limited here.

[0117] H22. Determine the foreground area and background area of ​​the two-dimensional image, and determine the edge grid area and the internal grid area among multiple grid areas.

[0118] The edge grid area is a grid area that has both foreground pixels and background pixels, and the internal grid area is a grid area that only contains foreground pixels.

[0119] Specifically, the method for dividing the foreground area and the background area in the two-dimensional image can refer to the above step S21, which will not be repeated here.

[0120] H23. Determine the density of feature points in the edge grid area and the internal grid area respectively.

[0121] Among them, the feature point density of the internal grid area is the preset baseline feature point density, the feature point density of the edge grid area is the product of the baseline feature point density and the foreground proportion, and the foreground proportion is the ratio of the number of foreground pixels in the edge grid area to the number of all pixels in the edge grid area.

[0122] In practical applications, a lower limit for the feature point density of each grid area can be set based on prior data (for example, at least 10 feature points are required in each grid area), and this lower limit of the feature point density is used as the baseline feature point density. In an embodiment of the present invention, the edge grid area is specifically the grid area where the edge of the foreground image is located. In this grid area, the foreground image only occupies a portion of the area, so there is no need to insert as many supplementary feature points. In this embodiment of the present invention, the corresponding feature point density is dynamically matched according to the proportion of foreground pixels in the edge grid area, making the insertion of supplementary feature points more uniform and accurate.

[0123] H24. Generate a local neighborhood with each visual feature point in the foreground area as the center. The local neighborhood is a two-dimensional Gaussian kernel whose radiation value decays from the center to the surrounding edges.

[0124] H25. Accumulate the radiation values ​​generated by each local neighborhood for any pixel point in the foreground area to obtain the cumulative radiation values ​​corresponding to each pixel point in the foreground area, and determine the mean of the cumulative radiation values ​​in the foreground area as the recognition threshold.

[0125] The specific contents of step H24 and step H25 can refer to the above-mentioned step S22 and step S23, which will not be repeated here.

[0126] H26. Based on the recognition threshold, determine whether each edge grid area and each internal grid area needs to be supplemented with feature points.

[0127] Specifically, since the number of foreground pixels in the edge grid area and the internal grid area is different (the embodiment of the present invention assumes that all the pixels in the internal grid area are foreground pixels, while the foreground pixels in the edge grid area only account for a certain proportion of them), and the foreground pixels are the effective processing objects in the embodiment of the present invention, whether a grid area needs to be supplemented with feature points needs to be judged in combination with the recognition threshold and the foreground proportion of the grid area (or the number of foreground pixels in the grid area). For example, assuming that the recognition threshold of a single pixel (i.e., the mean of the cumulative radiation value) is 10, if there are 100 foreground pixels in the current grid area (edge ​​grid area or internal grid area), 10*100 is used as the judgment threshold for each grid area, that is, if the sum of the cumulative radiation values ​​of all pixels in the current grid area is greater than 10*100, then no feature point supplementation is required. If it is less than or equal to 10*100, then it is confirmed that the current grid area needs to be supplemented with feature points.

[0128] H27. Based on the density of feature points corresponding to the edge grid area and the internal grid area, random insertion of supplementary feature points is performed on the grid area that needs to be supplemented with feature points.

[0129] For example, each grid area is traversed. If the sum of the cumulative radiation values ​​of all pixels in the current grid area is less than its corresponding judgment threshold, then based on the feature point density corresponding to the grid area, supplementary feature points are randomly inserted into the grid area (for example, based on the feature point density, it is determined that the grid area needs to have 20 visual feature points. Assuming that the grid area currently has 3 visual feature points, 17 supplementary feature points are randomly inserted into the remaining foreground pixels). Specifically, for edge grid areas, the random insertion of supplementary feature points can be further limited to the foreground pixels in the edge grid area to ensure the validity of the supplementary feature points.

[0130] It should be noted that the description of “feature point insertion” in the embodiments of the present invention refers to setting pixels that were originally non-feature points as feature points, rather than adding new pixels.

[0131] In an embodiment of the present invention, multiple grid areas are divided. When determining whether feature point supplementation is required, each grid area is identified independently, so that supplementary feature points are randomly inserted into each grid area, so that the distribution of supplementary feature points in the foreground area of ​​the entire two-dimensional image is more uniform.

[0132] The following describes the point cloud registration method based on fusion features. Figure 7 and Figure 9 , Figure 7 A flow chart of a point cloud registration method according to an embodiment of the present invention is shown. Figure 9 A schematic diagram of the architecture of a point cloud registration method in an embodiment of the present invention is shown. The point cloud registration method includes steps R1 to R6.

[0133] R1. Determine a first fused feature value corresponding to a first mapping feature point in the first point cloud and a second fused feature value corresponding to a second mapping feature point in the second point cloud.

[0134] Among them, the first point cloud is the source point cloud, and the second point cloud is the target point cloud. The embodiment of the present invention needs to find the transformation matrix that aligns the source point cloud to the target point cloud. The transformation matrix includes a rotation matrix and a translation vector that aligns the source point cloud to the target point cloud.

[0135] Specifically, in step R1 of the embodiment of the present invention, the above-mentioned method of fusing visual features with geometric features can be used to determine the fusion feature values ​​corresponding to the mapped feature points in the point cloud, which will not be described in detail here.

[0136] R2. Determine the nearest neighbor of each first mapping feature point.

[0137] Specifically, the following method is used to determine the nearest neighbor of each first mapping feature point: select any unprocessed first mapping feature point in the first point cloud, determine the first feature similarity between the first mapping feature point and each second mapping feature point, and take the second mapping feature point corresponding to the highest first feature similarity as the nearest neighbor of the first mapping feature point.

[0138] Exemplarily, the feature similarity between the first mapping feature point and the second mapping feature point is determined using the following relationship:

[0139]

[0140] In the formula, S(i,j) represents the feature similarity of feature point pair (i,j), i represents any first mapping feature point, j represents any second mapping feature point, Indicates the jth weight value of the current first mapping feature point i in the 1*n visual weight, Indicates the jth weight value in the 1*n geometric weight, S 2d(i, j) represents the similarity of visual features between the first point cloud and the second point cloud, S 3d (i, j) represents the geometric feature similarity between the first point cloud and the second point cloud, D 2d (i) represents the visual feature value of the first mapping feature point, D 2d (j) represents the visual feature value of the second mapping feature point, D 3d (i) represents the geometric eigenvalue of the first mapping feature point, D 3d (j) represents the geometric feature value of the second mapping feature point.

[0141] R3. Determine the nearest neighbor of each second mapping feature point.

[0142] Specifically, the following method is used to determine the nearest neighbor of each second mapping feature point: select any unprocessed second mapping feature point in the second point cloud, determine the second feature similarity between the second mapping feature point and each of the first mapping feature points, and use the first mapping feature point corresponding to the highest second feature similarity as the nearest neighbor of the second mapping feature point.

[0143] R4. Determine a set of feature point pairs of the first point cloud and the second point cloud based on the nearest neighbors of each first mapped feature point.

[0144] In one embodiment, feature point pairs of the first point cloud and the second point cloud can be determined based only on the nearest neighbors of each first mapping feature point (i.e., one-way matching). This embodiment can be used for more feature point pairs for registration, but the overall effectiveness of the feature point pairs is relatively low.

[0145] In another embodiment, the mapped feature points that are the nearest neighbors of each other in the first point cloud and the second point cloud can be selected as feature point pairs (bidirectional matching). This embodiment can use relatively fewer feature point pairs for registration, but the bidirectional matching can effectively eliminate one-way erroneous matches caused by factors such as feature instability or noise interference, ensuring that the feature point pairs have higher bidirectional consistency and matching reliability.

[0146] R5. Construct a compatibility graph based on the set of feature point pairs, search for the maximum cluster in the compatibility graph, and obtain the maximum cluster set.

[0147] A compatibility graph is a graph structure used to describe the matching relationship of point clouds. The initial set of feature point pairs is modeled as a graph to represent the geometric compatibility relationship between feature point pairs.

[0148] A first-order graph (FOG) of a compatibility graph consists of nodes and edges. Each node in the compatibility graph represents a feature point pair, that is, a correspondence between two point clouds. If two feature point pairs are geometrically compatible (i.e., the relative position and orientation between the two feature point pairs satisfy certain constraints), an edge is established between the two nodes, and the edge weight represents the strength of this compatibility.

[0149] For example, the establishment of an edge between two nodes in the compatibility graph can refer to the following relationship:

[0150]

[0151] Where c a Represents a node in the compatibility graph (i.e., a feature point pair (p i ,q j )), p i represents a point in the first point cloud, q j represents a point in the second point cloud, c b Represents another node in the compatibility graph, d ab Represents node c a To node c n The distance between them, σ represents a constant, S(c a ,c b ) represents node c a To node c b The weight of the edge, when S(c a ,c b ) table is greater than a certain threshold, node c a To node c b An edge will be formed between them. ab Calculation, specifically:

[0152] For the feature point pair c a , calculate the vector

[0153] For the feature point pair c b , calculate the vector

[0154] vector With vector The Euclidean distance between them can be obtained as the distance d ab .

[0155] After constructing the first-order graph of the compatibility graph, we can further construct the second-order graph (SOG) of the compatibility graph. The second-order graph considers the number of common compatible matches in the global set of feature point pairs. The weight matrix W of the second-order graph can be calculated with reference to the following relationship:

[0156] W=W FOG ⊙W SOG

[0157] Where W FOG Represents the weight matrix of the first-order graph (i.e., the above S(c a ,c b )), W SOG Represents the second-order compatibility measure matrix, and the “⊙” matrix is ​​element-wise multiplied.

[0158] About W SOG Calculation: For feature point pair c a and c b , and count the number of compatible matches they have in common in the global set of matches. This can be done by c a and c b Calculate the number of other matching point pairs that are compatible with each other to determine, according to c a and c b The number of commonly compatible matches, constructing the weight matrix W SOG .

[0159] See also Figure 8 , Figure 8 A schematic diagram of constructing a maximal clique in an embodiment of the present invention is shown. After constructing the second-order graph of the compatibility graph, a maximal clique is searched in the second-order graph of the compatibility graph to obtain a maximal clique set in the compatibility graph. Specifically: given an undirected graph G, clique C = (V,, E,), is a subset of G where any two nodes are connected by an edge, that is, a maximal clique is a clique that cannot be expanded by adding any nodes. For example, Figure 8 As shown in the figure, the lines of the same color in the left image form a maximum cluster. For example, the green line segments in the figure form a maximum cluster C1 = (c1, c2, c5, c 10 、c 11 ), the yellow line segments in the figure form a maximum cluster C2 = (c3, c4, c5, c6), and the blue line segments in the figure form a maximum cluster C3 = (c8, c9, c 10 ).

[0160] R6. Perform singular value decomposition on each maximal clique in the maximal clique set to obtain the posture hypothesis corresponding to each maximal clique, and select the optimal one among multiple posture hypotheses as the transformation matrix for point cloud registration between the first point cloud and the second point cloud.

[0161] Singular Value Decomposition (SVD) is a matrix decomposition method that can decompose any m×n matrix A into three specific matrix products (A=UΣV T), which are the orthogonal matrix U, the diagonal matrix Σ and the orthogonal matrix V. In point cloud registration, it is necessary to estimate a rigid body transformation matrix (including the rotation matrix and the translation vector) so that one point cloud can be aligned to another point cloud. Among the three matrices obtained by singular value decomposition, the orthogonal matrix U and the orthogonal matrix V can be used for the rotation matrix R, such as R = VU T Furthermore, the rotation matrix R can be used to obtain the translation vector t.

[0162] In the set of maximal clusters obtained in step R5 above, each maximal cluster represents a set of potential solutions for point cloud matching. By performing singular value decomposition on the maximal clusters, a set of 6-DOF pose hypotheses can be obtained. Specifically, the registration between the two point clouds satisfies the following relationship:

[0163] p'=R*p+t

[0164] Where R is the rotation matrix, t is the translation vector, and p and p' are the coordinates of the points in point clouds P and p' respectively.

[0165] The above-mentioned 6-DOF posture hypothesis is a set of (R, t). For the 6-DOF posture hypotheses corresponding to multiple maximal clusters in the maximal cluster set, the 6-DOF posture hypothesis with the best comprehensive score can be obtained through evaluation in several dimensions (such as mean absolute error, mean square error and inlier ratio), which can be used as the transformation matrix for point cloud registration between the first point cloud and the second point cloud.

[0166] The embodiments of the present invention have the following technical effects:

[0167] First, in point cloud registration, the geometric consistency between feature point pairs is an important basis for determining whether they are correctly matched. The process of constructing a maximal clique is actually to find a set of feature point pairs with highly consistent geometric features in the feature point pair set. These feature point pairs are usually compatible with each other in space, that is, the relative position and orientation relationship between the feature point pairs conforms to the constraints of rigid transformation. Second, three-dimensional point clouds are often affected by factors such as noise, occlusion, and data loss. The process of constructing a maximal clique can effectively filter out some invalid feature point pairs by mining local consistency information, thereby improving the robustness of the registration. Third, because the feature point pairs in the maximal clique are highly consistent, the generated pose hypothesis is more likely to be correct. By generating pose hypotheses from the maximal clique, the quality of the hypothesis can be significantly improved, thereby improving the accuracy of the registration.

[0168] See also Figure 10 , Figure 10 A schematic diagram of an electronic device according to an embodiment of the present invention is shown. The electronic device includes a memory 1100 and a processor 1200.

[0169] The processor 1200 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0170] The memory 1100 may include various types of storage units, such as system memory, read-only memory (ROM), and permanent storage devices. ROM may store static data or instructions required by the processor 1200 or other modules of the computer. The permanent storage device may be a readable and writable storage device. The permanent storage device may be a non-volatile storage device that retains stored instructions and data even when the computer is powered off. In some embodiments, the permanent storage device uses a large-capacity storage device (such as a magnetic or optical disk, flash memory) as the permanent storage device. In other embodiments, the permanent storage device may be a removable storage device (such as a floppy disk, optical drive). The system memory may be a readable and writable storage device or a volatile readable and writable storage device, such as dynamic random access memory. The system memory may store some or all instructions and data required by the processor during operation. In addition, the memory 1100 may include any combination of computer-readable storage media, including various types of semiconductor memory chips (DRAM, SRAM, SDRAM, flash memory, programmable read-only memory), and magnetic disks and / or optical disks may also be used. In some embodiments, the memory 1100 may include a readable and / or writable removable storage device, such as a compact disc (CD), a read-only digital versatile disc (e.g., DVD-ROM, double-layer DVD-ROM), a read-only Blu-ray disc, an ultra-density optical disc, a flash memory card (e.g., SD card, mini SD card, Micro-SD card, etc.), a magnetic floppy disk, etc. Computer-readable storage media do not include carrier waves and transient electronic signals transmitted wirelessly or wired.

[0171] The memory 1100 stores executable code for executing the method for fusing visual features and geometric features or the point cloud registration method in the above method embodiment. When the executable code is processed by the processor 1200, the processor 1200 can execute part or all of the methods described above.

[0172] In addition, the present invention also provides a computer-readable storage medium, such as a chip, a CD, etc., which stores an execution program. When the execution program is executed, it implements the above-mentioned visual feature and geometric feature fusion method or point cloud registration method.

[0173] It should be noted that the computer-readable storage medium described in the embodiments of the present disclosure is not limited to the embodiments given above, and can also be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the embodiments of the present disclosure, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or device.

[0174] It will be understood by those skilled in the art that, under the premise of no conflict, the above-mentioned preferred embodiments can be freely combined and superimposed. Among them, the flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, which contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions. The numbering of each step in this article is only for the convenience of description and reference, and is not used to limit the order of execution. The specific execution order is determined by the technology itself, and those skilled in the art can determine various allowable and reasonable orders based on the technology itself.

[0175] It should be noted that the use of step numbers (letters or numbers) to refer to certain specific method steps in the present invention is solely for the purpose of descriptive convenience and brevity, and is in no way intended to limit the order of these method steps. Those skilled in the art will appreciate that the order of the relevant method steps is determined by the technology itself and should not be unduly limited by the presence of step numbers. Those skilled in the art can determine various permissible and reasonable step orders based on the technology itself.

[0176] Those skilled in the art will appreciate that, provided there is no conflict, the above preferred solutions can be freely combined and superimposed.

[0177] It should be understood that the above-mentioned embodiments are merely illustrative and non-restrictive. Without departing from the basic principles of the present invention, various obvious or equivalent modifications or substitutions that can be made by those skilled in the art to the above-mentioned details will be included in the scope of the claims of the present invention.

Claims

1. A method for fusing visual features and geometric features, characterized in that: This method is applied to point cloud registration scenarios, including: Acquire a set of visual feature points of a two-dimensional image, the set of visual feature points including visual feature points and their corresponding visual feature values; acquire a set of geometric feature points of a three-dimensional point cloud corresponding to the two-dimensional image, the set of geometric feature points including geometric feature points and their corresponding geometric feature values; Supplementing feature points in an area of ​​the two-dimensional image where visual feature points are relatively sparse to obtain a set of enhanced feature points, wherein the set of enhanced feature points includes visual feature points and supplemented feature points; Determine feature points whose pixel coordinates in the enhanced feature point set and the geometric feature point set correspond to each other as mapping feature points; Performing visual feature encoding on the mapped feature points in the two-dimensional image to obtain a visual encoding value corresponding to each of the mapped feature points; determining a visual weight and a geometric weight corresponding to each of the mapped feature points based on the normalized visual encoding value, wherein the visual encoding value of a supplementary feature point among the mapped feature points is zero, and the sum of the visual weight and the geometric weight is a unit matrix; and The fusion feature value of each mapping feature point is determined based on the visual weight, the visual feature value, the geometric weight, and the geometric feature value corresponding to each mapping feature point.

2. The method for fusing visual features and geometric features according to claim 1, characterized in that: The supplementing of feature points in an area of ​​the two-dimensional image where visual feature points are relatively sparse includes: Obtaining visual feature points corresponding to a foreground area in the two-dimensional image; Generate a local neighborhood with each of the visual feature points as the center, wherein the local neighborhood is a two-dimensional Gaussian kernel whose radiation value decays from the center to the surrounding edges; Accumulating the radiation values ​​generated by each of the local neighborhoods for any pixel in the foreground area to obtain the cumulative radiation values ​​corresponding to each pixel in the foreground area, and determining the average of the cumulative radiation values ​​in the foreground area as the recognition threshold; Traversing each pixel in the foreground area based on the recognition threshold, and screening out N pixel points whose cumulative radiation values ​​are less than the recognition threshold; and Based on a preset feature point density, M pixels are randomly selected from the N pixels as supplementary feature points, where M is an integer less than N.

3. The method for fusing visual features and geometric features according to claim 1, characterized in that: The supplementing of feature points in an area of ​​the two-dimensional image where visual feature points are relatively sparse includes: Performing grid division on the two-dimensional image to obtain a plurality of grid areas; Determining a foreground area and a background area of ​​the two-dimensional image, and determining an edge grid area and an internal grid area among the multiple grid areas, wherein the edge grid area is a grid area having both foreground pixels and background pixels, and the internal grid area is a grid area containing only foreground pixels; Determining feature point densities of the edge grid area and the inner grid area, respectively, wherein the feature point density of the inner grid area is a preset baseline feature point density, the feature point density of the edge grid area is the product of the foreground ratio and the baseline feature point density, and the foreground ratio is the ratio of the number of foreground pixels in the edge grid area to the total number of pixels in the edge grid area; Generate a local neighborhood with each of the visual feature points in the foreground area as the center, wherein the local neighborhood is a two-dimensional Gaussian kernel whose radiation value decays from the center to the surrounding edges; Accumulating the radiation values ​​generated by each of the local neighborhoods for any pixel in the foreground area to obtain the cumulative radiation values ​​corresponding to each pixel in the foreground area, and determining the average of the cumulative radiation values ​​in the foreground area as the recognition threshold; Determining whether feature points need to be supplemented for each of the edge grid areas and each of the internal grid areas based on the recognition threshold; and Based on the density of the feature points corresponding to the edge grid area and the internal grid area, random insertion of supplementary feature points is performed on the grid area where feature points need to be supplemented.

4. The method for fusing visual features and geometric features according to claim 1, characterized in that: The performing visual feature encoding on the mapped feature points in the two-dimensional image includes: Perform the following processing on each visual feature point in the visual feature point set to obtain a visual encoding value of each visual feature point: Obtain 8 neighborhood pixels of the current visual feature point; compare the grayscale values ​​of the current visual feature point with those of each of the neighborhood pixels, mark the neighborhood pixels whose grayscale values ​​are greater than or equal to the current visual feature point as 1, and mark the neighborhood pixels whose grayscale values ​​are less than the current visual feature point as 0; arrange the mark values ​​of the 8 neighborhood pixels in order into a group of binary numbers; in the binary numbers, the situation where any two adjacent bits have different values ​​is regarded as a jump, and if the number of jumps is less than or equal to 2, set the visual coding value according to the number marked as 1 in the binary number; if the number of jumps is greater than 2, set the visual coding value to a preset value.

5. The method for fusing visual features and geometric features according to claim 4, characterized in that: The determining of the visual weight and the geometric weight corresponding to each of the mapping feature points based on the normalized visual coding value includes: Obtaining first visual coding values ​​of m mapped feature points in the first point cloud and second visual coding values ​​of n mapped feature points in the second point cloud respectively; and For any mapping feature point i among the m mapping feature points, the following processing is performed: the first visual coding value of the current mapping feature point i is summed with the n second visual coding values ​​and the average is taken to obtain a 1*n weight matrix; the weight matrix is ​​normalized to obtain the visual weight; the 1*n unit matrix is ​​subtracted from the visual weight to obtain the geometric weight, and the visual weight and the geometric weight are both 1*n normalized matrices.

6. The method for fusing visual features and geometric features according to claim 5, characterized in that: The normalizing process of the weight matrix includes: For any weight value in the 1*n weight matrix, normalization is performed based on the following relationship: Where W 2d (i, j) represents the weight value of the current normalization process, j is any one of the n second visual coding values, W min is the one with the smallest visual encoding value among the n weight values, W max is the one with the largest visual encoding value among the n weight values, is the normalized visual weight value.

7. The method for fusing visual features and geometric features according to claim 1, characterized in that: The determining of the fusion feature value of each mapping feature point based on the visual weight, the visual feature value, the geometric weight, and the geometric feature value corresponding to each mapping feature point includes: The fusion feature value of any of the mapping feature points is determined based on the following relationship: Where D f is the fusion feature value, is the visual weight, D 2d is the visual feature value, is the geometric weight, D 3d is the geometric characteristic value.

8. A point cloud registration method based on fusion features, characterized in that: include: Determine a first fused feature value corresponding to a first mapping feature point in the first point cloud and a second fused feature value corresponding to a second mapping feature point in the second point cloud using the fusion method according to claims 1 to 7; Determining the nearest neighbor of each of the first mapping feature points by using the following method: selecting any unprocessed first mapping feature point in the first point cloud, determining a first feature similarity between the first mapping feature point and each of the second mapping feature points, and selecting the second mapping feature point corresponding to the second mapping feature point with the highest first feature similarity as the nearest neighbor of the first mapping feature point; determining a set of feature point pairs of the first point cloud and the second point cloud based on the nearest neighbors of each of the first mapped feature points; constructing a compatibility graph based on the set of feature point pairs, searching for a maximal clique in the compatibility graph, and obtaining a maximal clique set; as well as Perform singular value decomposition on each of the maximal clusters in the maximal cluster set to obtain a posture hypothesis corresponding to each of the maximal clusters, and select the optimal one among the posture hypotheses as the transformation matrix for point cloud registration between the first point cloud and the second point cloud.

9. The point cloud registration method according to claim 8, characterized in that: Before determining a set of feature point pairs of the first point cloud and the second point cloud based on the nearest neighbors of each of the first mapped feature points, the method further includes: Determining the nearest neighbor of each second mapping feature point by using the following method: selecting any unprocessed second mapping feature point in the second point cloud, determining the second feature similarity between the second mapping feature point and each first mapping feature point, and using the first mapping feature point corresponding to the one with the highest second feature similarity as the nearest neighbor of the second mapping feature point; The determining, based on the nearest neighbors of each of the first mapped feature points, a set of feature point pairs of the first point cloud and the second point cloud, includes: Mapping feature points that are nearest neighbors of each other in the first point cloud and the second point cloud are selected as feature point pairs, and a feature point pair set of the first point cloud and the second point cloud is determined.

10. The point cloud registration method according to claim 8, characterized in that: The method further comprises: The feature similarity between the first mapping feature point and the second mapping feature point is determined using the following relationship: Where S(i,j) represents the feature similarity of the feature point pair (i,j), i represents any of the first mapping feature points, and j represents any of the second mapping feature points. Indicates the jth weight value of the current first mapping feature point i in the 1*n visual weight, Indicates the jth weight value in the 1*n geometric weight, S 2d (i, j) represents the visual feature similarity between the first point cloud and the second point cloud, S 3d (i, j) represents the geometric feature similarity between the first point cloud and the second point cloud, D 2d (i) represents the visual feature value of the first mapping feature point i, D 2d (j) represents the visual feature value of the second mapping feature point j, D 3d (i) represents the geometric eigenvalue of the first mapping feature point i, D 3d (j) represents the geometric eigenvalue of the second mapping feature point j.

11. An electronic device comprising a processor storing a computer program, characterized in that: When the computer program is executed by a processor, it implements the method for fusing visual features and geometric features as described in any one of claims 1 to 7, or the point cloud registration method based on fused features as described in any one of claims 8 to 10.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, it implements the method for fusing visual features and geometric features as described in any one of claims 1 to 7, or the point cloud registration method based on fused features as described in any one of claims 8 to 10.

Images