Vision-based point cloud image fusion method

Abstract

The invention relates to the technical field of image data processing, and discloses a point cloud image fusion method based on vision. Comprising the steps of extracting geometric features and visual features from point cloud data and image data, performing feature matching on point cloud feature points and image feature points, importing the geometric features and the visual features into fusion points in a model architecture, constructing an initial three-dimensional model, and performing optimization processing on the initial three-dimensional model to generate a high-quality three-dimensional model. Compared with the prior art, the method has the advantages that the matching feature points are screened after the matching operation of the point cloud feature points and the image feature points, the fusion precision of the point cloud data and the image data is improved, the probability of mismatching between the point cloud data and the image data is reduced, and the fusion accuracy of the point cloud data and the image data is improved by fusing the geometric features and the visual features. According to the method, detail information such as shapes, surfaces, colors and textures of target objects in various different types of scenes can be comprehensively embodied, so that a high-quality three-dimensional model is constructed.

Description

Technical Field

[0001] The present invention relates to the technical field of image data processing, and more specifically, to a vision-based point cloud image fusion method. Background Art

[0002] With the rapid development of computer vision and 3D reconstruction technology, vision-based point cloud image fusion methods have shown great application potential in many fields such as autonomous driving, robot navigation and industrial design. Since point cloud data lacks color and texture, and image data is difficult to directly express the 3D geometric structure of an object, there are limitations in using point cloud data or image data alone. Therefore, it is necessary to fuse point cloud data and image data through point cloud image fusion methods to generate a high-quality 3D model with both precise geometric shape and color texture information.

[0003] The patent application with reference publication number CN116579967A discloses a three-dimensional point cloud image fusion method and system based on computer vision, which can perform decision analysis on the point cloud image fusion method through a fusion analysis module, obtain decision coefficients by comprehensively analyzing and calculating various processing parameters of the fusion task, and improve the efficiency of image fusion processing by executing corresponding fusion tasks at appropriate fusion times. The point cloud image fusion processing process can be optimized and analyzed through a segmentation optimization module, so as to obtain a center value, a grid value and a random value through the optimization coefficients of fusion features processed by different segmentation modes, and adjust the selection weight of the image segmentation mode according to the center value, the grid value and the random value, so as to improve the efficiency of image segmentation processing;

[0004] Existing point cloud image fusion methods evaluate and analyze the accuracy of data matching and fusion by calculating the reliability of the fusion matching between point cloud data and image data. For example, in the above-mentioned patent application, the decision coefficient is calculated after comprehensive analysis of various processing parameters involved in the fusion task in the point cloud data and image data, and the fusion timing of the fusion task is determined by the decision coefficient to achieve the matching and fusion operation between the point cloud data and the image data. However, since the point cloud data and the image data need to process and calculate a large number of matching data of different types of feature points when matching, when a large number of different types and similar feature points need to be matched and calculated, it is easy for the feature point matching calculation to fail, which leads to low matching accuracy between the point cloud data and the image data, and cannot provide more accurate and reliable data support for the subsequent recognition and judgment of the target object.

[0005] In view of this, the present invention proposes a vision-based point cloud image fusion method to solve the above problems. Summary of the invention

[0006] In order to overcome the above defects of the prior art and to achieve the above objectives, the present invention provides the following technical solutions: a point cloud image fusion method based on vision, applied to a fusion server, comprising:

[0007] S1: acquiring point cloud data of the target object, and extracting geometric features from the preprocessed point cloud data after preprocessing the point cloud data, wherein the geometric features include shape features, surface features, and edge features;

[0008] S2: acquiring image data of the target object, and extracting visual features from the preprocessed image data after preprocessing the image data, wherein the visual features include color features, texture features, and boundary features;

[0009] S3: Identify the point cloud feature points of the point cloud data and the image feature points of the image data, perform feature matching on the point cloud feature points and the image feature points based on the matching criteria, and select the registration feature points;

[0010] S4: Build a model architecture based on point cloud data, and import geometric features and visual features into the fusion points in the model architecture to build an initial 3D model;

[0011] S5: After optimizing the initial three-dimensional model, a high-quality three-dimensional model is generated.

[0012] Furthermore, the preprocessing method of point cloud data is:

[0013] Import all point cloud data into the PCL library and load it, set the neighborhood size and outlier ratio, and determine the neighborhood of each point cloud data corresponding to the point location;

[0014] In the same neighborhood, measure the distance between any two points one by one, and add up all the distance values ​​and average them to get the point distance value. The points with a point distance value greater than the outlier ratio are recorded as outliers, and the point cloud data corresponding to the outliers are removed.

[0015] The remaining point cloud data is converted into a grid structure through grid filtering technology, and the standard radius of the grid structure is set. The center point of the grid structure is identified through computer vision, and a circle is drawn with the center point as the base point and the standard radius as the radius to generate a grid circle.

[0016] The points outside the grid circle are recorded as out-of-circle points, and the point cloud data corresponding to the out-of-circle points are eliminated to obtain the preprocessed point cloud data.

[0017] Furthermore, the method for extracting shape features, surface features and edge features is as follows:

[0018] A scale space is constructed, and Gaussian blurred images of different scales are generated through Gaussian blur processing and downsampling processing;

[0019] In Gaussian blurred images of different scales, the differences between Gaussian blurred images of adjacent scales are calculated one by one to obtain Gaussian difference images, and the gray values ​​of the points in the Gaussian difference images are marked, and the points corresponding to the maximum and minimum gray values ​​are recorded as extreme points;

[0020] Calculate the variance of the grayscale values ​​of all points in the area where the extreme point is located, record it as contrast, and remove the extreme points whose contrast is less than the preset contrast threshold, and record the remaining extreme points as key points;

[0021] The Sobel edge detection algorithm is used to calculate the gradients of the remaining points in the region where the keyword is located in the X-axis direction and the Y-axis direction, and the gradient direction and magnitude are obtained. The horizontal axis represents the gradient direction, and the vertical axis represents the gradient magnitude, and a histogram is generated.

[0022] Select the gradient direction with the largest peak in the histogram as the main direction of the key point, and combine it with the amplitude of the key point to generate a feature descriptor;

[0023] The keywords of the feature descriptors are identified one by one, and the feature descriptors whose keywords are shape, surface and edge are recorded as shape feature, surface feature and edge feature respectively.

[0024] Furthermore, the image data preprocessing method is:

[0025] Import all image data into the graphics processor, and identify the center dividing line of all image data through computer vision technology;

[0026] The tilt angle of the image data is continuously adjusted by a tilt correction algorithm until the central segmentation line of the image data coincides with the standard segmentation line, thereby generating corrected image data;

[0027] The noise of the corrected image data is removed by a median filtering algorithm, and the edge information of the denoised image data is enhanced by a Laplace algorithm to generate preprocessed image data.

[0028] Furthermore, the color feature, texture feature and boundary feature extraction method are as follows:

[0029] The preprocessed image data is converted from the RGB color space to the HSV color space, and the image data is divided into A adjacent sub-regions using the color automatic segmentation technology. One of the color components of the HSV color space is used as the index of the sub-region, and the image data is converted into a binary color index set to generate color features;

[0030] A pixel point is randomly selected from A sub-regions as the target point, and the pixel values ​​of the target point and the pixel points in the sub-regions adjacent to the target point are compared in a clockwise direction;

[0031] When the pixel value of the target point is greater than or equal to the pixel value of the pixel point in the annular adjacent sub-region, the pixel point in the annular adjacent sub-region is assigned a value of 1;

[0032] When the pixel value of the target point is less than the pixel value of the pixel in the annular adjacent sub-region, the pixel in the annular adjacent sub-region is assigned 0 to generate a binary number, and the frequencies of 0 and 1 in A sub-regions are calculated in turn to generate A histograms, and the A histograms are connected to generate texture features;

[0033] The preprocessed image data is subjected to edge detection by the Canny algorithm to obtain an edge image, and the boundary direction and boundary curvature of the edge image are identified by computer vision technology. After combining the boundary direction and boundary curvature, the boundary feature is obtained.

[0034] Furthermore, the matching criterion is: the smaller the Euclidean distance between the geometric feature point and the image feature point is, the higher the matching degree between the geometric feature point and the image feature point is.

[0035] Furthermore, the feature matching method of point cloud feature points and image feature points is:

[0036] The rotation matrix and translation vector between the lidar and the high-resolution camera are pre-calibrated, and the feature points of the point cloud are converted from the lidar coordinate system to the camera coordinate system using the rotation matrix and translation vector as the conversion standard to generate the feature point coordinates in the camera coordinate system;

[0037] The expression of the feature point in the camera coordinate system is:

[0038] ;

[0039] In the formula, is the coordinate of the feature point in the camera coordinate system, is the rotation matrix, is the coordinate of the feature point in the radar coordinate system, is the translation vector;

[0040] The camera intrinsic parameter matrix is ​​pre-calibrated, and the feature point coordinates in the camera coordinate system are converted into image pixel coordinates through the camera intrinsic parameter matrix;

[0041] The expression of image pixel coordinates is:

[0042] ;

[0043] In the formula, is the image pixel coordinate, is the camera intrinsic parameter matrix;

[0044] The Euclidean distance formula is used to calculate the Euclidean distances between the point cloud feature points and the image feature points in the multidimensional space one by one, and the point cloud feature points and the image feature points corresponding to the minimum value of the Euclidean distance are recorded as matching feature points, and the matching feature points are matched.

[0045] Furthermore, the method for screening the registration feature points is:

[0046] The error threshold and the number of iterations of the RANSAC algorithm are pre-set, and a blank set is initialized, which is recorded as the internal point set;

[0047] All matching feature points are used as initial samples in turn, the transformation matrix is ​​estimated through the initial samples, and the distance from the midpoint of the transformation matrix to the transformation plane is calculated, which is recorded as the projection error;

[0048] The matching feature points corresponding to the initial samples whose projection errors are less than the error threshold are recorded as inliers, and the inliers are imported into the inlier set, and the number of inliers in the inlier set is counted to obtain the initial value;

[0049] The remaining matching feature points are recorded as verification samples in turn, the projection error of the verification samples is calculated, the inliers are identified from the remaining matching feature points, and the identified inliers are imported into the inlier set;

[0050] Count the number of inliers in the inlier set in real time to obtain a real-time value. When the real-time value is greater than the initial value, update the transformation matrix and the inlier set.

[0051] Repeat the above steps until the preset number of iterations is reached or the real-time value no longer increases, select the transformation matrix corresponding to the maximum value of the real-time value as the optimal transformation matrix, and record the inliers of the inliers set of the optimal transformation matrix as the registration feature points.

[0052] Furthermore, the initial three-dimensional model is constructed as follows:

[0053] Based on the point cloud data, a model architecture corresponding to the point cloud data is constructed through 3D reconstruction technology. The original points corresponding to the point cloud data are marked one by one in the model architecture, and the original points are numbered in ascending order according to the order of marking to generate fusion points.

[0054] According to the numbering from small to large, the shape features, surface features, edge features, color features, texture features and boundary features corresponding to the registration feature points are sequentially fused through the data fusion algorithm to obtain fused features;

[0055] The fusion features are assigned to the corresponding fusion points one by one, and the point cloud of the assigned fusion points is colored to construct the initial three-dimensional model.

[0056] Furthermore, the method for generating a high-quality three-dimensional model is as follows:

[0057] The boundary of the initial three-dimensional model is identified by computer vision technology, and a line is drawn along the location of the boundary to generate a model boundary line;

[0058] The fusion points located on the boundary line of the model are recorded as key points, and the geometric features and visual features on the key points are recorded as key features;

[0059] The key features are enhanced by a feature enhancement algorithm to generate detail features, and the grid where the detail features are located is recorded as a detail area;

[0060] The initial 3D model is regionally simplified using a model simplification algorithm, and detail regions and detail features are restored using detail recovery technology to generate a high-quality 3D model.

[0061] Technical effects and advantages of a vision-based point cloud image fusion method of the present invention:

[0062] By extracting geometric features from point cloud data and visual features from image data, the present invention can provide direct and clear matching objects for the matching of point cloud data and image data, and screen matching feature points after the matching operation of point cloud feature points and image feature points, which not only improves the fusion accuracy of point cloud data and image data, but also greatly reduces the probability of mismatching between the two. By fusing geometric features with visual features, the shape, surface, color, texture and other detail information of target objects in various types of scenes can be fully and accurately reflected, thereby constructing a high-quality three-dimensional model that can truly and accurately reflect the point cloud information and image information of the target object, ensuring that the fused three-dimensional model has stronger details, higher clarity and better realism, thereby providing more accurate and reliable data support for subsequent recognition and judgment of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic diagram of a flow chart of a vision-based point cloud image fusion method provided in Embodiment 1 of the present invention;

[0064] Figure 2 A schematic diagram of a module of a vision-based point cloud image fusion system provided in Embodiment 2 of the present invention;

[0065] Figure 3 A schematic diagram of the structure of an electronic device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0066] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0067] Example 1: Please refer to Figure 1 As shown, the vision-based point cloud image fusion method described in this embodiment is applied to a fusion server, including:

[0068] S1: acquiring point cloud data of the target object, and after preprocessing the point cloud data, extracting geometric features from the preprocessed point cloud data;

[0069] The target object refers to a three-dimensional object that needs to be subjected to point cloud image fusion operation, and serves as an object that provides the basis and original data for point cloud image fusion. The point cloud data is a data set that is obtained after scanning the target object with a scanning device and can represent the coordinates of each point of the target object in three-dimensional space, thereby serving as one of the data for subsequent point cloud image fusion. Specifically, the scanning device includes but is not limited to a three-dimensional scanner, a stereo camera, and a lidar. In this embodiment, the point cloud data is obtained after scanning the target object with a lidar.

[0070] Since the point cloud data is obtained by scanning the target image directly with a scanning device, the amount of point cloud data is relatively large, and the noise of the point cloud data is also problematic. In order to improve the subsequent use of the point cloud data, the point cloud data needs to be preprocessed to ensure the high quality and consistency of the point cloud data.

[0071] The preprocessing method of point cloud data is:

[0072] Import all point cloud data into the PCL library and load it, set the neighborhood size and outlier ratio, and determine the neighborhood of the area where each point cloud data corresponds to. The neighborhood size is used to calculate the number of points in the neighborhood of each point, and the outlier ratio is the threshold for judging whether a point is an outlier. Usually, the outlier ratio is calculated based on the mean and standard deviation of the distance of the points in the neighborhood, so that those points that are too far away in the neighborhood can be eliminated to reduce the amount of point cloud data.

[0073] In the same neighborhood, measure the distance between any two points one by one, and add up all the distance values ​​and average them to get the point distance value. The points with a point distance value greater than the outlier ratio are recorded as outliers, and the point cloud data corresponding to the outliers are removed.

[0074] The remaining point cloud data is converted into a grid structure through grid filtering technology, and the standard radius of the grid structure is set. The center point of the grid structure is identified through computer vision, and a circle is drawn with the center point as the base point and the standard radius as the radius to generate a grid circle; the standard radius is used to numerically represent the maximum scanning radius of the grid structure and provide a basis for the subsequent identification of points outside the circle;

[0075] The points outside the grid circle are recorded as out-of-circle points, and the point cloud data corresponding to the out-of-circle points are eliminated to obtain the preprocessed point cloud data.

[0076] The preprocessed point cloud data can maintain high quality and consistency, but the point cloud data at this time contains a large amount of information and features, and the types are also more complex. In order to accurately and quickly obtain direct information for subsequent point cloud image fusion from the point cloud data, it is necessary to extract geometric features from the preprocessed point cloud data so that the geometric features can clearly represent the shape, size, boundary and other features of the target object in the point cloud data;

[0077] Geometric features include shape features, surface features and edge features; shape features refer to features in point cloud data that can reflect the overall or local geometric shape of the target object; surface features refer to geometric features on the surface of the target object in point cloud data, including but not limited to details such as protrusions, depressions, and textures; edge features refer to features on the surface or structural boundaries of the target object in point cloud data;

[0078] The extraction methods of shape features, surface features and edge features are:

[0079] A scale space is constructed, and Gaussian blurred images of different scales are generated through Gaussian blur processing and downsampling processing; Gaussian blur processing is to reduce noise and details and highlight the main features. In high-dimensional space, this can be achieved by applying Gaussian blur to each dimension of the point cloud data separately. Downsampling can be achieved through methods such as voxel grid downsampling to reduce the number of data points while maintaining the main shape and structure of the point cloud data;

[0080] In Gaussian blurred images of different scales, the differences between Gaussian blurred images of adjacent scales are calculated one by one to obtain Gaussian difference images, and the gray values ​​of the points in the Gaussian difference images are marked, and the points corresponding to the maximum and minimum gray values ​​are recorded as extreme points;

[0081] Calculate the variance of the grayscale values ​​of all points in the area where the extreme point is located, record it as contrast, and remove the extreme points with contrast less than the preset contrast threshold, and record the remaining extreme points as key points; the preset contrast threshold is the minimum value of the variance of the grayscale values ​​of the points recorded as extreme points, which can provide a basis for the subsequent identification and judgment of extreme points;

[0082] The Sobel edge detection algorithm is used to calculate the gradients of the remaining points in the region where the keyword is located in the X-axis direction and the Y-axis direction, and the gradient direction and magnitude are obtained. The horizontal axis represents the gradient direction, and the vertical axis represents the gradient magnitude, and a histogram is generated.

[0083] Select the gradient direction with the largest peak in the histogram as the main direction of the key point, and combine it with the amplitude of the key point to generate a feature descriptor;

[0084] The keywords of the feature descriptors are identified one by one, and the feature descriptors with the keywords of shape, surface and edge are recorded as shape features, surface features and edge features respectively. The keywords are used to concisely represent the true meaning of the descriptors, so that shape features, surface features and edge features can be identified quickly and accurately.

[0085] S2: acquiring image data of the target object, and after preprocessing the image data, extracting visual features from the preprocessed image data;

[0086] Image data refers to a data set that is obtained after photographing a target object with a camera device and can represent the color, texture and detail information of the target object, providing a high-quality image data source for subsequent image processing and three-dimensional reconstruction; specifically, the image data in this embodiment is obtained by photographing the target object from multiple angles with a high-resolution camera.

[0087] Since the image data is obtained by directly shooting the target image with a camera, the image data quality and clarity may have certain defects. In order to improve the use effect of the subsequent image data, it is necessary to pre-process the image data;

[0088] The preprocessing method of image data is:

[0089] All image data are imported into the graphics processor, and the center dividing line of all image data is identified through computer vision technology; the center dividing line refers to a line passing through the center point of the image data and bisecting the image data in the vertical direction, and serves as a basis for determining whether the image data is tilted;

[0090] The tilt angle of the image data is continuously adjusted by the tilt correction algorithm until the center dividing line of the image data coincides with the standard dividing line, thereby generating the corrected image data; the standard dividing line refers to the center dividing line of the image data when it is kept vertical and not tilted, and serves as the basis for subsequent adjustment of the real-time tilt state of the image data;

[0091] The noise of the corrected image data is removed by a median filtering algorithm, and the edge information of the denoised image data is enhanced by a Laplace algorithm to generate preprocessed image data.

[0092] The preprocessed image data can maintain a high level of clarity and contrast, but the information contained in the preprocessed image data is of complex types and large in quantity. In order to accurately and concisely represent the preprocessed image data, it is necessary to extract visual features from the preprocessed image data so that the visual features can represent the feature information about the visual direction in the preprocessed image data;

[0093] Visual features include color features, texture features, and boundary features; color features refer to features in image data that can reflect the overall or local color information of the target object, texture features refer to features in image data that can reflect the boundary direction information of the target object, and boundary features refer to features in image data that can reflect the boundary line distribution information of the target object;

[0094] The extraction methods of color features, texture features and boundary features are as follows:

[0095] The preprocessed image data is converted from the RGB color space to the HSV color space, and the image data is divided into A adjacent sub-regions using the color automatic segmentation technology. One of the color components of the HSV color space is used as the index of the sub-region, and the image data is converted into a binary color index set to generate color features;

[0096] A pixel point is randomly selected from A sub-regions as the target point, and the pixel values ​​of the target point and the pixel points in the sub-regions adjacent to the target point are compared in a clockwise direction; the clockwise direction can ensure the orderliness of the comparison of the pixel points in the sub-regions adjacent to the target point, avoiding the phenomenon of chaotic and disordered comparison process;

[0097] When the pixel value of the target point is greater than or equal to the pixel value of the pixel point in the annular adjacent sub-region, the pixel point in the annular adjacent sub-region is assigned a value of 1;

[0098] When the pixel value of the target point is less than the pixel value of the pixel in the annular adjacent sub-region, the pixel in the annular adjacent sub-region is assigned 0 to generate a binary number, and the frequencies of 0 and 1 in A sub-regions are calculated in turn to generate A histograms, and the A histograms are connected to generate texture features;

[0099] The preprocessed image data is edge detected by the Canny algorithm to obtain an edge image, and the boundary direction and boundary curvature of the edge image are identified by computer vision technology. The boundary direction and boundary curvature are combined to obtain the boundary feature. The boundary direction is used to represent the distribution direction of the boundary position of the image data, and the boundary curvature is used to represent the curvature of the boundary position of the image data, so that the boundary features of the image data can be accurately represented.

[0100] S3: Identify the point cloud feature points of the point cloud data and the image feature points of the image data, perform feature matching on the point cloud feature points and the image feature points based on the matching criteria, and select the registration feature points;

[0101] Point cloud feature points refer to the points where geometric features are located in the point cloud data, so that each point cloud feature point has a unique and independent feature descriptor. Since point cloud feature points are relatively prominent and obvious points in the point cloud data, the key points in the point cloud data are point cloud feature points.

[0102] Image feature points refer to the points in the image data where the visual features are located, so that each image feature point has a unique and independent feature descriptor. Since image feature points are relatively prominent and obvious points in the image data, the target points in the image data are image feature points.

[0103] After obtaining the point cloud feature points and image feature points, since the point cloud feature points are the point representation of the target object in the point cloud, and the image feature points are the point representation of the target object in the image, in order to screen and match the point cloud feature points and the image feature points with high similarity, it is necessary to perform matching operations on the point cloud feature points and the image feature points, and realize effective matching of the point cloud feature points and the image feature points under the restriction of the matching criteria, and then find the points with extremely high similarity in the point cloud data and the image data;

[0104] The matching criterion is: the smaller the Euclidean distance between the geometric feature points and the image feature points, the higher the matching degree between the geometric feature points and the image feature points; that is, the geometric feature points in the point cloud data and the image feature points in the image data can be accurately matched;

[0105] The feature matching method of point cloud feature points and image feature points is:

[0106] The rotation matrix and translation vector between the lidar and the high-resolution camera are pre-calibrated, and the point cloud feature points are converted from the radar coordinate system to the camera coordinate system using the rotation matrix and translation vector as the conversion standard to generate the feature point coordinates in the camera coordinate system; the rotation matrix and translation vector are used as external parameters between the lidar and the high-resolution camera to provide the point cloud feature points with information for rotation and translation, and are used to realize the coordinate conversion between the radar coordinate system and the camera coordinate system;

[0107] The expression of the feature point in the camera coordinate system is:

[0108] ;

[0109] In the formula, is the coordinate of the feature point in the camera coordinate system, is the rotation matrix, is the coordinate of the feature point in the radar coordinate system, is the translation vector;

[0110] The camera intrinsic parameter matrix is ​​pre-calibrated, and the feature point coordinates in the camera coordinate system are converted into image pixel coordinates through the camera intrinsic parameter matrix;

[0111] The expression of image pixel coordinates is:

[0112] ;

[0113] In the formula, is the image pixel coordinate, is the camera intrinsic parameter matrix;

[0114] The Euclidean distance formula is used to calculate the Euclidean distances between the point cloud feature points and the image feature points in the multidimensional space one by one, and the point cloud feature points and the image feature points corresponding to the minimum value of the Euclidean distance are recorded as matching feature points, and the matching feature points are matched.

[0115] After feature matching between the point cloud feature points and the image feature points, the Euclidean distance between the point cloud feature points and the image feature points after feature matching may not be the minimum, which will cause errors in the matching results. In order to avoid the phenomenon of matching errors, it is necessary to verify and screen the point cloud feature points and the image feature points after feature matching to obtain the registration feature points.

[0116] The screening method for registration feature points is:

[0117] The error threshold and the number of iterations of the RANSAC algorithm are set in advance, and a blank set is initialized, which is recorded as the internal point set. By setting the error threshold and the number of iterations of the RANSAC algorithm, the matching verification and screening of the registration feature points can be provided with an operational basis to ensure that the registration feature points can be safely and quickly screened.

[0118] All matching feature points are used as initial samples in turn, the transformation matrix is ​​estimated through the initial samples, and the distance from the midpoint of the transformation matrix to the transformation plane is calculated, which is recorded as the projection error;

[0119] The matching feature points corresponding to the initial samples whose projection errors are less than the error threshold are recorded as inliers, and the inliers are imported into the inlier set, and the number of inliers in the inlier set is counted to obtain the initial value;

[0120] The remaining matching feature points are recorded as verification samples in turn, the projection error of the verification samples is calculated, the inliers are identified from the remaining matching feature points, and the identified inliers are imported into the inlier set;

[0121] Count the number of inliers in the inlier set in real time to obtain a real-time value. When the real-time value is greater than the initial value, update the transformation matrix and the inlier set.

[0122] Repeat the above steps until the preset number of iterations is reached or the real-time value no longer increases, select the transformation matrix corresponding to the maximum value of the real-time value as the optimal transformation matrix, and record the inliers of the inliers set of the optimal transformation matrix as the registration feature points.

[0123] Through the continuous iterative calculation of the above-mentioned RANSAC algorithm, the matching results of the point cloud feature points and the image feature points after feature matching can be verified, and the feature points with low matching accuracy can be eliminated to ensure the high accuracy of feature matching between the point cloud feature points and the image feature points.

[0124] S4: Build a model architecture based on point cloud data, and import geometric features and visual features into the fusion points in the model architecture to build an initial 3D model;

[0125] The model architecture is a 3D simulation model architecture of the target object that is initially constructed based on point cloud data and has only geometric features but no visual features. It serves as the basis for the subsequent construction of the initial 3D model and can provide a reasonable model for the import of geometric features and visual features.

[0126] The fusion point refers to the data fusion position in the model architecture where the position and information correspondence can be achieved when the point cloud data and image data are fused, and serves as the node of the subsequent initial 3D model to ensure that the point cloud data and image data remain accurate and reasonable when fused, thereby improving the accuracy of the subsequent initial 3D model;

[0127] The initial 3D model refers to the 3D simulation model of the target object constructed by preliminary fusion of geometric features and visual features, and serves as the basis for subsequent optimization and recognition of the target object;

[0128] The initial 3D model is constructed as follows:

[0129] Based on the point cloud data, a model architecture corresponding to the point cloud data is constructed through 3D reconstruction technology. The original points corresponding to the point cloud data are marked one by one in the model architecture, and the original points are numbered in ascending order according to the order of marking to generate fusion points. The model architecture is a basic model without any substantial meaning, and serves as the basis for the construction of the subsequent initial 3D model. The original points are points in the model architecture used to provide positions for the fusion of point cloud data and image data, and serve as the predecessor of fusion points.

[0130] According to the numbering from small to large, the shape features, surface features, edge features, color features, texture features and boundary features corresponding to the registration feature points are sequentially fused through the data fusion algorithm to obtain fused features;

[0131] The fusion features are assigned to the corresponding fusion points one by one, and the point cloud is colored after the fusion points are assigned, and the initial 3D model is constructed. After the fusion points are colored, each point corresponding to the point cloud data can be assigned a corresponding color, so that the initial 3D model can have a relatively consistent appearance color with the target object.

[0132] It should be noted that the constructed initial three-dimensional model can perform an initial and reasonable three-dimensional simulation on the point cloud data and image data of the target object, ensuring that the target object can be displayed by the initial three-dimensional simulation.

[0133] S5: After optimizing the initial three-dimensional model, a high-quality three-dimensional model is generated;

[0134] Since the initial 3D model is only an initialized 3D simulation model of the target object, the initial 3D model may have a rough surface, unclear texture, and prominent details, which makes it impossible for the initial 3D model to accurately and realistically perform 3D simulation of the target object. Therefore, it is necessary to optimize the initial 3D model to improve the overall performance of the initial 3D model, so as to obtain a high-quality 3D model.

[0135] The method for generating high-quality 3D models is:

[0136] The boundary of the initial three-dimensional model is identified by computer vision technology, and a line is drawn along the location of the boundary to generate a model boundary line;

[0137] The fusion points located on the boundary line of the model are recorded as key points, and the geometric features and visual features on the key points are recorded as key features;

[0138] The key features are enhanced by the feature enhancement algorithm to generate detail features, and the grid where the detail features are located is recorded as the detail area; by generating detail features and detail areas, the detail information with important significance and obvious features in the three-dimensional model can be enhanced and highlighted, so that the three-dimensional model can be more accurately compared with the target object, and the detail information of the target object can be represented more accurately;

[0139] The initial 3D model is regionally simplified using a model simplification algorithm, and detail regions and detail features are restored using detail recovery technology to generate a high-quality 3D model.

[0140] It should be noted that in addition to detail enhancement processing on the initial three-dimensional model, surface smoothing and texture mapping operations can also be performed on the initial three-dimensional model according to actual optimization needs. In the surface smoothing operation, in order to eliminate the noise and unevenness generated in the data fusion process, the Kalman filtering method is used for dynamic smoothing processing, which effectively improves the smoothness of the surface of the initial three-dimensional model and improves the realism of the initial three-dimensional model. In order to deal with the sampling problem in texture mapping, bilinear interpolation technology is used to improve the accuracy and effect of texture mapping, so as to improve the realism and visual effect of the initial three-dimensional model. Surface smoothing operations and texture mapping operations are both existing technologies in this field, and related algorithms are also widely used.

[0141] In this embodiment, by extracting geometric features from point cloud data and visual features from image data, direct and clear matching objects can be provided for the matching of point cloud data and image data, and matching feature points are screened after the matching operation of point cloud feature points and image feature points, which not only improves the fusion accuracy of point cloud data and image data, but also greatly reduces the probability of mismatching between the two. By fusing geometric features and visual features, the shape, surface, color, texture and other detailed information of target objects in various types of scenes can be fully and accurately reflected, thereby constructing a high-quality three-dimensional model that can truly and accurately reflect the point cloud information and image information of the target object, ensuring that the fused three-dimensional model has stronger details, higher clarity, and better realism, thereby providing more accurate and reliable data support for subsequent identification and judgment of the target object.

[0142] Example 2: Please refer to Figure 2 As shown, the part not described in detail in this embodiment is described in the first embodiment, and a point cloud image fusion system based on vision is provided, which is applied to a fusion server and is used to implement a point cloud image fusion method based on vision, including a first feature extraction module, a second feature extraction module, a feature matching module, a data fusion module and a model optimization module, wherein each module is connected through a wired or wireless network;

[0143] A first feature extraction module is used to obtain point cloud data of a target object, and after preprocessing the point cloud data, extract geometric features from the preprocessed point cloud data, where the geometric features include shape features, surface features, and edge features;

[0144] A second feature extraction module is used to obtain image data of the target object, and after preprocessing the image data, extract visual features from the preprocessed image data, the visual features including color features, texture features and boundary features;

[0145] A feature matching module is used to identify the point cloud feature points of the point cloud data and the image feature points of the image data, perform feature matching on the point cloud feature points and the image feature points based on the matching criteria, and select the registration feature points;

[0146] The data fusion module is used to build a model framework based on point cloud data, and import geometric features and visual features into the fusion points in the model framework to build an initial 3D model;

[0147] The model optimization module is used to generate a high-quality three-dimensional model after optimizing the initial three-dimensional model.

[0148] Example 3: Please refer to Figure 3 As shown, this embodiment discloses an electronic device, including a processor and a memory;

[0149] Wherein, the memory stores a computer program that can be called by the processor;

[0150] The processor executes the vision-based point cloud image fusion method by calling the computer program stored in the memory.

[0151] Since the electronic device introduced in this embodiment is an electronic device used to implement a vision-based point cloud image fusion method in Embodiment 1 of this application, based on the vision-based point cloud image fusion method introduced in the embodiment of this application, a person skilled in the art can understand the specific implementation of the electronic device of this embodiment and its various variations, so how the electronic device implements the method in the embodiment of this application is not described in detail here. As long as a person skilled in the art implements an electronic device used in a vision-based point cloud image fusion method in the embodiment of this application, it belongs to the scope of protection of this application.

[0152] The above description is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be covered by the protection scope of the present invention.

Claims

1. A vision-based point cloud image fusion method, applied to a fusion server, characterized in that: include: S1: acquiring point cloud data of the target object, and extracting geometric features from the preprocessed point cloud data after preprocessing the point cloud data, wherein the geometric features include shape features, surface features, and edge features; S2: acquiring image data of the target object, and extracting visual features from the preprocessed image data after preprocessing the image data, wherein the visual features include color features, texture features, and boundary features; S3: Identify the point cloud feature points of the point cloud data and the image feature points of the image data, perform feature matching on the point cloud feature points and the image feature points based on the matching criteria, and select the registration feature points; S4: Build a model architecture based on point cloud data, and import geometric features and visual features into the fusion points in the model architecture to build an initial 3D model; S5: After optimizing the initial three-dimensional model, a high-quality three-dimensional model is generated.

2. The method for point cloud image fusion based on vision according to claim 1, characterized in that: The preprocessing method of point cloud data is: Import all point cloud data into the PCL library and load it, set the neighborhood size and outlier ratio, and determine the neighborhood of each point cloud data corresponding to the point location; In the same neighborhood, measure the distance between any two points one by one, and add up all the distance values ​​and average them to get the point distance value. The points with a point distance value greater than the outlier ratio are recorded as outliers, and the point cloud data corresponding to the outliers are removed. The remaining point cloud data is converted into a grid structure through grid filtering technology, and the standard radius of the grid structure is set. The center point of the grid structure is identified through computer vision, and a circle is drawn with the center point as the base point and the standard radius as the radius to generate a grid circle. The points outside the grid circle are recorded as out-of-circle points, and the point cloud data corresponding to the out-of-circle points are eliminated to obtain the preprocessed point cloud data.

3. The method for point cloud image fusion based on vision according to claim 2, characterized in that: The extraction methods of shape features, surface features and edge features are: A scale space is constructed, and Gaussian blurred images of different scales are generated through Gaussian blur processing and downsampling processing; In Gaussian blurred images of different scales, the differences between Gaussian blurred images of adjacent scales are calculated one by one to obtain Gaussian difference images, and the gray values ​​of the points in the Gaussian difference images are marked, and the points corresponding to the maximum and minimum gray values ​​are recorded as extreme points; Calculate the variance of the grayscale values ​​of all points in the area where the extreme point is located, record it as contrast, and remove the extreme points whose contrast is less than the preset contrast threshold, and record the remaining extreme points as key points; The Sobel edge detection algorithm is used to calculate the gradients of the remaining points in the region where the keyword is located in the X-axis direction and the Y-axis direction, and the gradient direction and magnitude are obtained. The horizontal axis represents the gradient direction, and the vertical axis represents the gradient magnitude, and a histogram is generated. Select the gradient direction with the largest peak in the histogram as the main direction of the key point, and combine it with the amplitude of the key point to generate a feature descriptor; The keywords of the feature descriptors are identified one by one, and the feature descriptors whose keywords are shape, surface and edge are recorded as shape feature, surface feature and edge feature respectively.

4. The method for point cloud image fusion based on vision according to claim 3, characterized in that: The image data preprocessing method is: Import all image data into the graphics processor, and identify the center dividing line of all image data through computer vision technology; The tilt angle of the image data is continuously adjusted by a tilt correction algorithm until the central segmentation line of the image data coincides with the standard segmentation line, thereby generating corrected image data; The noise of the corrected image data is removed by a median filtering algorithm, and the edge information of the denoised image data is enhanced by a Laplace algorithm to generate preprocessed image data.

5. The method for point cloud image fusion based on vision according to claim 4, characterized in that: The extraction methods of color features, texture features and boundary features are as follows: The preprocessed image data is converted from the RGB color space to the HSV color space, and the image data is divided into A adjacent sub-regions using the color automatic segmentation technology. One of the color components of the HSV color space is used as the index of the sub-region, and the image data is converted into a binary color index set to generate color features; A pixel point is randomly selected from A sub-regions as the target point, and the pixel values ​​of the target point and the pixel points in the sub-regions adjacent to the target point are compared in a clockwise direction; When the pixel value of the target point is greater than or equal to the pixel value of the pixel point in the annular adjacent sub-region, the pixel point in the annular adjacent sub-region is assigned a value of 1; When the pixel value of the target point is less than the pixel value of the pixel in the annular adjacent sub-region, the pixel in the annular adjacent sub-region is assigned 0 to generate a binary number, and the frequencies of 0 and 1 in A sub-regions are calculated in turn to generate A histograms, and the A histograms are connected to generate texture features; The preprocessed image data is subjected to edge detection by the Canny algorithm to obtain an edge image, and the boundary direction and boundary curvature of the edge image are identified by computer vision technology. After combining the boundary direction and boundary curvature, the boundary feature is obtained.

6. The method for point cloud image fusion based on vision according to claim 5, characterized in that: The matching criterion is: the smaller the Euclidean distance between the geometric feature point and the image feature point, the higher the matching degree between the geometric feature point and the image feature point.

7. The method for vision-based point cloud image fusion according to claim 6, characterized in that: The feature matching method of point cloud feature points and image feature points is: The rotation matrix and translation vector between the lidar and the high-resolution camera are pre-calibrated, and the feature points of the point cloud are converted from the lidar coordinate system to the camera coordinate system using the rotation matrix and translation vector as the conversion standard to generate the feature point coordinates in the camera coordinate system; The expression of the feature point in the camera coordinate system is: ； In the formula, is the coordinate of the feature point in the camera coordinate system, is the rotation matrix, is the coordinate of the feature point in the radar coordinate system, is the translation vector; The camera intrinsic parameter matrix is ​​pre-calibrated, and the feature point coordinates in the camera coordinate system are converted into image pixel coordinates through the camera intrinsic parameter matrix; The expression of image pixel coordinates is: ； In the formula, is the image pixel coordinate, is the camera intrinsic parameter matrix; The Euclidean distance formula is used to calculate the Euclidean distances between the point cloud feature points and the image feature points in the multidimensional space one by one, and the point cloud feature points and the image feature points corresponding to the minimum value of the Euclidean distance are recorded as matching feature points, and the matching feature points are matched.

8. The method for vision-based point cloud image fusion according to claim 7, characterized in that: The screening method for registration feature points is: The error threshold and the number of iterations of the RANSAC algorithm are pre-set, and a blank set is initialized, which is recorded as the internal point set; All matching feature points are used as initial samples in turn, the transformation matrix is ​​estimated through the initial samples, and the distance from the midpoint of the transformation matrix to the transformation plane is calculated, which is recorded as the projection error; The matching feature points corresponding to the initial samples whose projection errors are less than the error threshold are recorded as inliers, and the inliers are imported into the inlier set, and the number of inliers in the inlier set is counted to obtain the initial value; The remaining matching feature points are recorded as verification samples in turn, the projection error of the verification samples is calculated, the inliers are identified from the remaining matching feature points, and the identified inliers are imported into the inlier set; Count the number of inliers in the inlier set in real time to obtain a real-time value. When the real-time value is greater than the initial value, update the transformation matrix and the inlier set. Repeat the above steps until the preset number of iterations is reached or the real-time value no longer increases, select the transformation matrix corresponding to the maximum value of the real-time value as the optimal transformation matrix, and record the inliers of the inliers set of the optimal transformation matrix as the registration feature points.

9. The method for vision-based point cloud image fusion according to claim 8, characterized in that: The initial 3D model is constructed as follows: Based on the point cloud data, a model architecture corresponding to the point cloud data is constructed through 3D reconstruction technology. The original points corresponding to the point cloud data are marked one by one in the model architecture, and the original points are numbered in ascending order according to the order of marking to generate fusion points. According to the numbering from small to large, the shape features, surface features, edge features, color features, texture features and boundary features corresponding to the registration feature points are sequentially fused through the data fusion algorithm to obtain fused features; The fusion features are assigned to the corresponding fusion points one by one, and the point cloud of the assigned fusion points is colored to construct the initial three-dimensional model.

10. The method for point cloud image fusion based on vision according to claim 9, characterized in that: The method for generating high-quality 3D models is: The boundary of the initial three-dimensional model is identified by computer vision technology, and a line is drawn along the location of the boundary to generate a model boundary line; The fusion points located on the model boundary line are recorded as key points, and the geometric features and visual features on the key points are recorded as key features; The key features are enhanced by a feature enhancement algorithm to generate detail features, and the grid where the detail features are located is recorded as a detail area; The initial 3D model is regionally simplified using a model simplification algorithm, and detail regions and detail features are restored using detail recovery technology to generate a high-quality 3D model.

Images