Method and device for fusing two-dimensional image and point cloud based on multi-dimensional feature registration

Abstract

The application discloses a kind of based on multi-dimensional feature registration's fusion method of two-dimensional image and point cloud, comprising: pre-processing two-dimensional image and three-dimensional point cloud data;Feature extraction is carried out using convolutional neural network;Shared feature representation is obtained using convolution kernel full connection layer fusion;Deep learning model is constructed: deep learning model includes encoder and decoder, and encoder is mainly composed of convolution layer and pooling layer, and convolution layer includes multiple different scale convolution kernels;Full connection layer generates output feature;Decoder predicts semantic category for each pixel;Model training and optimization obtain prediction model;The two-dimensional image of the object to be processed and corresponding three-dimensional point cloud data are input into prediction model, and the fusion information of the object to be processed is obtained.The application has the beneficial effects of improving fusion accuracy and meeting the needs of high-precision fusion.The application also provides a kind of fusion device, comprising preprocessing module, registration projection module, feature extraction module, feature fusion prediction module and output module.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of fusion point cloud technology. More specifically, the present application relates to a two-dimensional image and point cloud fusion method and device based on multi-dimensional feature registration. BACKGROUND

[0002] Although three-dimensional point cloud data has real three-dimensional coordinates, it lacks real texture information. Under the premise of lacking prior knowledge, the human eye has very limited recognition ability for three-dimensional point clouds. Digital images have rich texture information, and the imaging effect is similar to what the human eye sees, which meets human cognition of the real world. The disadvantage is that it cannot directly represent the three-dimensional real world with two-dimensional. If three-dimensional point cloud and two-dimensional digital image can be fused, the overall information of the target surface can be obtained, including the real three-dimensional coordinates and rich texture information of the measured object surface. There are several fusion technologies at present:

[0003] 1. Three-dimensional point cloud and two-dimensional image alignment: three-dimensional point cloud and two-dimensional image are usually represented by different coordinate systems, so they need to be aligned to a shared coordinate first. This can be achieved through camera calibration and perspective geometry techniques, such as using a camera projection model to convert image coordinates to point cloud coordinates.

[0004] 2. Three-dimensional point cloud projection: three-dimensional point cloud can be projected onto a two-dimensional image plane by projecting three-dimensional points in the three-dimensional point cloud to corresponding two-dimensional image locations. Camera parameters and geometric relationships can be used for projection calculations. After projection, the attributes of three-dimensional point cloud (such as color and texture) can be matched with image pixels, thereby realizing the fusion of three-dimensional point cloud and two-dimensional image.

[0005] 3. Feature extraction of three-dimensional point cloud and two-dimensional image: features can be extracted from three-dimensional point cloud and two-dimensional image respectively, and fusion can be achieved by matching these features. For example, traditional feature extraction algorithms (such as SISF, SURF, etc.) are used in images to monitor key points and descriptors, and then corresponding projection techniques are used in point cloud to map these features to corresponding three-dimensional points, thereby establishing the correspondence between image and point cloud.

[0006] 4. Deep learning fusion of three-dimensional point cloud and two-dimensional image: deep learning methods have made great progress in both two-dimensional image processing and three-dimensional point cloud analysis. Deep learning models can be used to process two-dimensional images and three-dimensional point cloud data simultaneously, thereby realizing their fusion. For example, a neural network model can be designed to accept two-dimensional images and three-dimensional point clouds as input, and jointly learn the relationship between two-dimensional and three-dimensional features to solve various tasks such as object detection, scene understanding, etc.

[0007] 5. Sensor fusion: In some cases, multiple sensors (such as cameras, lidar, etc.) can be used to acquire both two-dimensional images and three-dimensional point cloud data simultaneously, and they can be fused together. For example, sensor fusion algorithms such as Extended Kalman Filter (EKF) or Particle Filter (PF) can be used to achieve more accurate and complete scene information.

[0008] However, the current two-dimensional image and three-dimensional point cloud fusion technology has the following shortcomings:

[0009] 1. Data inconsistency: Two-dimensional images and three-dimensional point clouds have different data representation methods. Two-dimensional images are represented in pixels, while three-dimensional point clouds are represented in point coordinates and attribute information. This causes data inconsistency, which requires data format conversion and alignment for effective fusion.

[0010] 2. Data registration problem: Fusing two-dimensional images and three-dimensional point clouds requires data registration, which means projecting two-dimensional images into three-dimensional space or mapping three-dimensional point clouds back to two-dimensional image planes. However, the registration process may introduce errors, leading to inaccurate fusion results. Solving this problem requires accurate sensor calibration and registration algorithms.

[0011] 3. Data sparsity and noise: Both two-dimensional images and three-dimensional point clouds have data sparsity and noise. Two-dimensional images may have occlusions, lighting changes, and texture blurring, while three-dimensional point clouds may have missing points and noise points. These problems affect the quality and accuracy of the fusion results and require data processing and noise filtering operations.

[0012] 4. Computational complexity: The processing of two-dimensional images and three-dimensional point clouds has different computational complexities. Two-dimensional image processing usually uses convolutional neural network-based methods. Therefore, effective fusion of two-dimensional images and three-dimensional point clouds needs to solve the problem of computational complexity mismatch.

[0013] 5. Cross-modal information fusion: Fusing two-dimensional images and three-dimensional point clouds requires considering cross-modal information fusion, i.e., how to effectively fuse color, texture, and shape information in two-dimensional images with set structure and attribute information in three-dimensional point clouds.

[0014] In summary, the current fusion method is single and has poor accuracy, which cannot meet the needs of some high-precision tasks. SUMMARY

[0015] An object of the present application is to solve at least the above problems and to provide at least the advantages described later.

[0016] In order to achieve the purposes and other advantages according to the present application, a method for fusing two-dimensional image and point cloud based on multi-dimensional feature registration is provided, comprising the following steps:

[0017] S1, obtaining two-dimensional image and corresponding three-dimensional point cloud data of an object, and respectively pre-processing;

[0018] S2, registration: converting the pre-processed two-dimensional image and three-dimensional point cloud data to the same coordinate system, and establishing the spatial position correlation between the two-dimensional image and the three-dimensional point cloud;

[0019] S3, point cloud projection: projecting the registered three-dimensional point cloud data into the corresponding two-dimensional image space;

[0020] S4, feature extraction: using convolutional neural network to extract features from the two-dimensional image and the three-dimensional point cloud data with mutual spatial position correlation, respectively, to obtain corresponding feature representations;

[0021] S5, fusing the feature representations in step S4 using a convolution kernel full connection layer to obtain a shared feature representation;

[0022] S6, constructing a deep learning model: the deep learning model includes an encoder and a decoder, wherein the encoder is mainly composed of a convolution layer and a pooling layer, the convolution layer includes a plurality of convolution kernels with different scales, each convolution kernel is convolved with the input data to retain spatial structure information and extract local features at the same time, the convolution kernel is the weight in the convolution operation, the convolution operation performs sliding window calculation on the input data and generates a specific feature map, a nonlinear activation function is used to activate the feature map to introduce nonlinear features; the pooling layer performs down-sampling on the feature map;

[0023] After multiple convolution layers and pooling layers, the input shared feature representation and the weight are multiplied by a full connection layer to perform matrix multiplication and nonlinear transformation, and the output feature is generated;

[0024] The decoder is mainly composed of a dedicated convolution layer and a convolution layer, the dedicated convolution layer up-samples the output feature to increase the resolution, and then classifies and predicts through the convolution layer to output an image with the same size as the input image and predict the semantic category for each pixel;

[0025] S7, model training and optimization: using the labeled training data set to train the deep learning model constructed in step S6, and performing feature descent optimization according to the loss function, after iterative training and verification, a prediction model is obtained;

[0026] S8, inputting the two-dimensional image and the corresponding three-dimensional point cloud data of the object to be processed into the prediction model in step S7 after processing by steps S1-S5 to obtain the fusion information of the object to be processed.

[0027] Preferably, the method of preprocessing in step S1 is specifically: preprocessing the two-dimensional image, including scaling to a target size, normalizing the two-dimensional image pixel range to 0-1, and cropping the two-dimensional image to obtain a region of interest.

[0028] The three-dimensional point cloud is denoised, a filter type is selected and filter parameters are set for filter processing, and a resampling method is selected and resampling parameters are set to process the three-dimensional point cloud data after filter processing.

[0029] Preferably, in the scaling process, an interpolation algorithm is used to process the differences between image pixels.

[0030] Preferably, the normalization method is to divide the pixel value by 255, so that the pixel range from 0 to 255 is mapped to between 0 and 1.

[0031] Preferably, the cropping standard is defined by the position, pixel value or bounding box of the image.

[0032] Preferably, the filter type includes one of a mean filter, a median filter and a Gaussian filter.

[0033] Preferably, the resampling method includes one of voxel gridding, nearest neighbor sampling and surface-based sampling.

[0034] Preferably, the pooling layer uses a max pooling method, and the maximum value of a certain region in the feature map is selected as the feature after downsampling.

[0035] Preferably, the activation function is a softmax function, which generates a probability distribution for each class.

[0036] A device for a two-dimensional image and point cloud fusion method based on multi-dimensional feature registration is provided, comprising:

[0037] A preprocessing module for obtaining a two-dimensional image and corresponding three-dimensional point cloud data of an object, and preprocessing them respectively;

[0038] A registration and projection module for converting the preprocessed two-dimensional image and three-dimensional point cloud data to the same coordinate system, establishing a spatial position association between the two-dimensional image and the three-dimensional point cloud, and projecting the registered three-dimensional point cloud data into the corresponding two-dimensional image space;

[0039] A feature extraction module for using a convolutional neural network to extract features from the two-dimensional image and the three-dimensional point cloud data with mutual spatial position association, respectively, to obtain corresponding feature representations;

[0040] The feature fusion prediction module is used to build a deep learning model. The deep learning model includes an encoder and a decoder. The encoder is mainly composed of a convolution layer and a pooling layer. The convolution layer includes multiple convolution kernels of different scales. Each convolution kernel performs a convolution operation with the input data, retaining spatial structure information and extracting local features at the same time. The convolution kernel is the weight in the convolution operation. The convolution operation performs a sliding window calculation on the input data and generates a specific feature map. The feature map is activated by a nonlinear activation function to introduce nonlinear features; the pooling layer downsamples the feature map;

[0041] After multiple convolutional and pooling layer processing, a fully connected layer is used to perform matrix multiplication and nonlinear transformation on the input shared feature representation and weights to generate output features;

[0042] The decoder is mainly composed of dedicated convolutional layers and convolutional layers. The dedicated convolutional layers upsample the output features to increase the resolution, and then pass them through the convolutional layers for classification prediction. The output is consistent with the input image size and predicts the semantic category for each pixel.

[0043] And for model training and optimization, the deep learning model constructed in step S6 is trained using the labeled training data set, and feature descent optimization is performed according to the loss function. After iterative training and verification, a prediction model is obtained;

[0044] The output module is used to input the two-dimensional image of the object to be processed and the corresponding three-dimensional point cloud data after being processed in steps S1 to S5 into the prediction model in step S7 to obtain the fusion information of the object to be processed.

[0045] The present invention includes at least the following beneficial effects: based on the deep learning of point clouds and images, the concept of network fusion model is introduced, an end-to-end deep learning network is designed, two-dimensional images and three-dimensional point cloud data are taken as input, and they are merged through network layer fusion operations to obtain an accurate mapping relationship between visible light images and three-dimensional point clouds, thereby obtaining more accurate and robust two-dimensional image and three-dimensional point cloud fusion results.

[0046] Other advantages, objectives and features of the present invention will be reflected in part from the following description and will be understood by those skilled in the art through study and practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a framework diagram of the fusion method of the present invention. DETAILED DESCRIPTION

[0048] The present invention will be described in further detail below in conjunction with the accompanying drawings so that those skilled in the art can implement the invention with reference to the description.

[0049] like Figure 1As shown, the present invention provides a method for fusing a two-dimensional image and a point cloud based on multi-dimensional feature registration, comprising the following steps:

[0050] S1. Obtain a two-dimensional image of the object and corresponding three-dimensional point cloud data, and preprocess them respectively. The preprocessing method specifically includes: preprocessing the two-dimensional image, including scaling it to a target size, normalizing the two-dimensional image pixel range to 0-1, and cropping the two-dimensional image to obtain a region of interest;

[0051] Among them, scaling is to adjust the size of an image to a different size. The scaling ratio can be set as needed. Generally, a scaling factor less than 1 can be used to reduce the image, while a scaling factor greater than 1 can be used to enlarge the image. The specific scaling factor depends on the original size and target size of the image. For example, if the original image size is 500×500 pixels, we can reduce it to half of the original size, that is, 250×250 pixels. Similarly, if we want to enlarge the image to twice the original size, we can resize it to 1000×1000 pixels. During the scaling process, interpolation algorithms (such as nearest neighbor interpolation, bilinear interpolation, or bicubic interpolation) can be used to handle the differences between pixels.

[0052] Normalization involves mapping the pixel values ​​of an image to a specific range (usually 0 to 1). This aims to eliminate differences in pixel values ​​between different images, making them more similar in scale and easier to compare and process. One normalization method that can be used is to divide the pixel values ​​by 255, mapping the pixel range from 0 to 255 to 0 to 1.

[0053] Cropping involves selecting a region of interest from an image based on specific criteria, retaining it, and removing other areas. Cropping criteria can be defined to suit different requirements and applications. For example, cropping criteria can be defined based on image location, pixel values, or bounding boxes. This allows you to capture images with specific regions, facilitating subsequent processing or analysis.

[0054] Denoise the 3D point cloud, select the filter type and set the filter parameters for filtering, and select the resampling method and set the resampling parameters to process the filtered 3D point cloud data.

[0055] The general steps of filtering include: defining the filter; selecting the appropriate filter type and parameters. Common filter types include mean filters, median filters, and Gaussian filters. The filter selection depends on the specific application requirements and image characteristics.

[0056] 3D point cloud resampling is the process of resampling 3D point cloud data to adjust the density and resolution of the point cloud or remove unnecessary noise points. After resampling, new 3D point cloud data can be obtained.

[0057] During the resampling of 3D point clouds, different algorithms and strategies can be used for processing. Resampling methods include voxel gridding, nearest neighbor sampling, and surface-based sampling. The data output after sampling is still a 3D point cloud data, but its characteristics vary depending on the resampling goal and algorithm. For example, if the resampling is to adjust the density of the point cloud, the output point cloud may have fewer or more points, and the density of the point cloud can be uniform or non-uniform in space. If the resampling is to remove noise, the output point cloud may reduce the number of noise points or adjust the shape of the point cloud to better represent the actual object.

[0058] The resampled 3D point cloud data can be used for further 3D point cloud processing and applications, such as object recognition, point cloud registration, and modeling. The output data is still a 3D point cloud, but its structure and characteristics will change depending on the resampling method and parameters. This depends on the resampling algorithm used and the parameter settings during the processing.

[0059] S2, Registration: Convert the pre-processed 2D image and 3D point cloud data into the same coordinate system and establish the spatial position association between the 2D image and the 3D point cloud;

[0060] S3, point cloud projection: project the registered 3D point cloud data into the corresponding 2D image space;

[0061] S4. Feature extraction: Convolutional neural networks are used to extract features from two-dimensional images and three-dimensional point cloud data that have mutual spatial position correlations to obtain corresponding feature representations;

[0062] The specific steps of feature extraction include:

[0063] Data preparation: First, you need to prepare the original dataset, which can be in the form of images, text, in, etc. Ensure the quality and applicability of the dataset.

[0064] Feature selection: Before feature extraction, feature selection can be performed. Feature selection involves selecting or filtering out the most discriminative and important features from the original dataset. This can reduce the feature dimensionality, improve computational efficiency, and avoid the influence of merit or noise features.

[0065] Feature Extraction Method Selection: Convolutional neural networks are used to extract features from spatially correlated 2D images and 3D point cloud data, generating corresponding feature representations. Feature extraction is performed using algorithms such as SISF, SURF, and FAST. Feature Representation: The extracted features require appropriate representation for subsequent processing and analysis. Vector representation is used to ensure consistency and comparability.

[0066] Feature evaluation and selection: After extracting features, the evaluation and selection of features are performed to understand the quality and contribution of features to the task. Correlation, information gain, variance, and other indicators are used to evaluate the importance of features.

[0067] Feature preprocessing: According to the requirements of the task and the nature of the features, necessary feature preprocessing operations are performed. For example, normalization, standardization, dimensionality reduction, and other operations can further optimize the representation and processing effect.

[0068] Among them, the following aspects of features are extracted: structural features: used to express and represent the structural information of data, such as set shape, topology and structure, hierarchical relationship, etc. In image processing, structural features can be edges, corner points, straight lines, etc. Statistical features: features extracted based on the statistical properties of data, including mean, variance, standard deviation, maximum and minimum value, histogram, etc. Statistical features are often used to describe the distribution and overall characteristics of data. Visual features: used for feature extraction of image and video data, including color features, texture features, shape features, SIFT (Scale-Invariant Feature Transform) descriptor, HOG (Histogram of Oriented Gradients), etc.

[0069] S5, the feature representation in step S4 is fused by a convolution kernel full connection layer to obtain a shared feature representation; after serial fusion connection, the following can be obtained: data / feature fusion result, feature representation enhancement, decision result fusion, information fusion result.

[0070] The specific steps of the convolution kernel full connection layer include:

[0071] Input: First, prepare the data set of two-dimensional images and three-dimensional point clouds (the feature extraction results of the two are shown above). The image is represented as a matrix, and the three-dimensional point cloud is represented as three-dimensional coordinate information and possible additional attributes.

[0072] Feature extraction and preprocessing: For two-dimensional images and three-dimensional point clouds, different feature extraction and preprocessing methods are applied. For images, convolutional neural networks (CNN) are used for feature extraction, and for three-dimensional point clouds, point cloud-specific algorithms (Pointnet++) are used for feature extraction. Local and global information of images and three-dimensional point clouds can be captured to generate corresponding feature representations.

[0073] Feature fusion: After feature extraction is completed, feature fusion operations can be performed. The features of images and three-dimensional point clouds are concatenated or superimposed. The specific operation is to connect the image and point cloud features into a larger feature vector, or to fuse them into a shared feature representation through a convolution kernel full connection layer. This time, the shared feature representation is selected.

[0074] S6, constructing a deep learning model: next, a deep learning model is constructed to learn the fused feature representation and relationship. In the deep learning model, convolution kernels are used for spatial and channel level extraction of features, and fully connected layers are used for mapping and combination of features. The convolution layer performs convolution operation on the input through sliding window to extract local features. The fully connected layer performs matrix multiplication and nonlinear transformation of the input features and weights to generate output features.

[0075] The deep learning model includes an encoder and a decoder, wherein the encoder is mainly composed of convolution layers and pooling layers, the convolution layers include a plurality of convolution kernels of different scales, each convolution kernel is convolved with the input data to preserve spatial structure information and extract local features at the same time, the convolution kernel is the weight in the convolution operation, the convolution operation calculates the input data through sliding window and generates a specific feature map, a nonlinear activation function is used to activate the feature map to introduce nonlinear features; the pooling layer down-samples the feature map; wherein different scale convolution kernels are introduced in the convolution neural network to process different size receptive fields. For example, different size convolution kernels or multi-scale dilated convolution are used. In this way, multiple scale details and context information can be captured and fused together to improve the feature expression ability.

[0076] After multiple convolution layers and pooling layers, the shared feature representation of the input is multiplied by the weight in the matrix and nonlinearly transformed by the fully connected layer to generate the output feature.

[0077] In the present application, the basic steps of the encoder include:

[0078] Input layer: accepts the input of shared feature representation.

[0079] Convolution layer: the convolution layer is the core part of CNN, which is composed of multiple convolution kernels. Each convolution kernel is convolved with the input data to extract local features. The convolution operation calculates the input data through sliding window to generate a feature map.

[0080] Activation function: after the convolution layer, a nonlinear activation function such as ReLU (Rectified Linear Unit) is used to activate the feature map to introduce nonlinear features.

[0081] Pooling layer: the pooling layer reduces the number of parameters and computational complexity by down-sampling the feature map while preserving important features. Max pooling is used to select the maximum value of a certain region in the feature map as the down-sampled feature.

[0082] Fully connected layer: After a series of convolutional and pooling layers, the features extracted by the fully connected layer are used for classification or regression. The fully connected layer flattens the features into vectors and generates the final result through matrix multiplication and activation functions.

[0083] Output layer: For classification tasks, a softmax function is used to generate a probability distribution for each category.

[0084] The decoder is mainly composed of dedicated convolutional layers and convolutional layers. The dedicated convolutional layers upsample the output features to increase the resolution, and then pass them through the convolutional layers for classification prediction. The output is consistent with the input image size and predicts the semantic category for each pixel.

[0085] S7, model training and optimization: Use the labeled training data set to train the deep learning model constructed in step S6, and perform feature descent optimization according to the loss function, iteratively train and verify, adjust the parameters and structure of the model, and obtain a prediction model;

[0086] Output: After training, the predictive model can be used to predict or generate new data. For a given input, the predictive model will generate the corresponding output, such as a classification label, regression value, or other desired results.

[0087] A device for a method for fusing a two-dimensional image and a point cloud based on multi-dimensional feature registration is provided, comprising:

[0088] A preprocessing module, which is used to obtain a two-dimensional image of an object and corresponding three-dimensional point cloud data, and perform preprocessing on each of them;

[0089] A registration and projection module is used to convert the pre-processed 2D image and 3D point cloud data into the same coordinate system, establish a spatial position association between the 2D image and the 3D point cloud; and project the registered 3D point cloud data into the corresponding 2D image space;

[0090] A feature extraction module is used to extract features from two-dimensional images and three-dimensional point cloud data that have a spatial relationship with each other using a convolutional neural network to obtain corresponding feature representations;

[0091] The feature fusion prediction module is used to build a deep learning model. The deep learning model includes an encoder and a decoder. The encoder is mainly composed of a convolution layer and a pooling layer. The convolution layer includes multiple convolution kernels of different scales. Each convolution kernel performs a convolution operation with the input data, retaining spatial structure information and extracting local features at the same time. The convolution kernel is the weight in the convolution operation. The convolution operation performs a sliding window calculation on the input data and generates a specific feature map. The feature map is activated by a nonlinear activation function to introduce nonlinear features; the pooling layer downsamples the feature map;

[0092] After the multi-convolution layer and the pooling layer processing, the input shared feature representation is multiplied by the weight using the full connection layer and is nonlinearly changed to generate the output feature;

[0093] The decoder is mainly composed of a professional convolution layer and a convolution layer. The professional convolution layer up-samples the output feature to increase the resolution, and then the convolution layer is used for classification prediction to output the semantic category for each pixel, which is consistent with the size of the input image;

[0094] And for model training and optimization, the labeled training dataset is used to train the deep learning model constructed in step S6, and the feature is optimized according to the loss function. After iterative training and verification, the prediction model is obtained;

[0095] An output module is configured to input the two-dimensional image and the corresponding three-dimensional point cloud data of the object to be processed into the prediction model in step S7 after processing by steps S1-S5, so as to obtain the fusion information of the object to be processed.

[0096] <application example 1>

[0097] Applied to the real scene of the substation:

[0098] Step 1, data acquisition: Obtain the two-dimensional image and three-dimensional point cloud data of the substation through unmanned aerial vehicles, cameras or laser scanners and other equipment. Two-dimensional images can provide color and texture information, while three-dimensional point clouds can provide geometric structure and spatial coordinate information.

[0099] Step 2, data registration: Register the two-dimensional image and three-dimensional point cloud data to ensure that they are in the same coordinate system. Registration can be achieved by using feature point matching, automatic calibration or manual calibration and other methods. The goal of registration is to correspond the three-dimensional point cloud of the two-dimensional image to the same spatial position, thereby establishing the association between them.

[0100] Step 3, point cloud projection: Project the registered three-dimensional point cloud data into the corresponding two-dimensional image space. This can be achieved by mapping the three-dimensional coordinates of each point to the corresponding pixel position. Projection can be completed using algorithms such as geometric conversion and camera model.

[0101] Step 4, feature extraction: Extract feature information from the registered and projected data. For two-dimensional images, computer vision techniques such as edge detection, feature descriptor extraction and other methods can be used to obtain the shape and texture information of the object. For three-dimensional point clouds, surface normal, curvature and other geometric features can be extracted, or feature extraction based on shape descriptors can be performed.

[0102] Step 4, data fusion: The feature representation is fused to obtain a shared feature representation using a convolution kernel full connection layer;

[0103] Step five, input the shared feature representation in step four into the prediction model to obtain the fusion information of the object to be processed;

[0104] Step five, visualization and application: visualize the fused two-dimensional image and three-dimensional point cloud data, and apply it to specific tasks and applications. Visualization can be achieved through rendering techniques such as point cloud rendering, texture mapping, etc. The fused data can be used in substation modeling, safety analysis, maintenance planning, etc.

[0105] Although the embodiments of the present application have been disclosed as above, they are not limited only to the applications listed in the specification and embodiments, and can be fully applied to various fields suitable for the present application, and additional modifications can be easily made by those skilled in the art, and therefore the present application is not limited to specific details and the figures shown and described herein, without departing from the general concept defined by the claims and the equivalent scope.

Claims

1. A fusion method of two-dimensional image and point cloud based on multi-dimensional feature registration, characterized in that: The following steps are involved: S1. Obtain a two-dimensional image of the object and the corresponding three-dimensional point cloud data, and pre-process them respectively; S2, Registration: Convert the pre-processed 2D image and 3D point cloud data into the same coordinate system and establish the spatial position association between the 2D image and the 3D point cloud; S3, point cloud projection: project the registered 3D point cloud data into the corresponding 2D image space; S4. Feature extraction: Convolutional neural networks are used to extract features from two-dimensional images and three-dimensional point cloud data that have mutual spatial position correlations to obtain corresponding feature representations; S5, the feature representation in step S4 is fused using a convolution kernel fully connected layer to obtain a shared feature representation; S6. Build a deep learning model: The deep learning model includes an encoder and a decoder. The encoder is mainly composed of convolutional layers and pooling layers. The convolutional layer includes multiple convolution kernels of different scales. Each convolution kernel performs a convolution operation with the input data, retaining spatial structure information and extracting local features at the same time. The convolution kernel is the weight in the convolution operation. The convolution operation performs a sliding window calculation on the input data and generates a specific feature map. The feature map is activated using a nonlinear activation function to introduce nonlinear features. The pooling layer downsamples the feature map. After multiple convolutional and pooling layer processing, a fully connected layer is used to perform matrix multiplication and nonlinear transformation on the input shared feature representation and weights to generate output features; The decoder is mainly composed of dedicated convolutional layers and convolutional layers. The dedicated convolutional layers upsample the output features to increase the resolution, and then pass them through the convolutional layers for classification prediction. The output is consistent with the input image size and predicts the semantic category for each pixel. S7, model training and optimization: Use the labeled training data set to train the deep learning model constructed in step S6, and perform feature descent optimization based on the loss function. After iterative training and verification, a prediction model is obtained; S8. Input the two-dimensional image of the object to be processed and the corresponding three-dimensional point cloud data processed in steps S1 to S5 into the prediction model in step S7 to obtain fusion information of the object to be processed.

2. The method for fusing two-dimensional images and point clouds based on multi-dimensional feature registration according to claim 1, wherein: The preprocessing method in step S1 is specifically as follows: preprocessing the two-dimensional image, including scaling to a target size, normalizing the pixel range of the two-dimensional image to 0-1, and cropping the two-dimensional image to obtain a region of interest; Denoise the 3D point cloud, select the filter type and set the filter parameters for filtering, and select the resampling method and set the resampling parameters to process the filtered 3D point cloud data.

3. The method for fusing two-dimensional images and point clouds based on multi-dimensional feature registration according to claim 2, wherein: During the scaling process, an interpolation algorithm is used to handle the differences between image pixels.

4. The method for fusing two-dimensional images and point clouds based on multi-dimensional feature registration according to claim 2, wherein: Normalization is done by dividing the pixel values ​​by 255 so that the pixel range from 0 to 255 is mapped to 0 to 1.

5. The method for fusing two-dimensional images and point clouds based on multi-dimensional feature registration according to claim 2, wherein: Define cropping criteria in terms of image position, pixel value, or bounding box.

6. The method for fusing two-dimensional images and point clouds based on multi-dimensional feature registration according to claim 2, wherein: The filter type includes one of a mean filter, a median filter, and a Gaussian filter.

7. The method for fusing two-dimensional images and point clouds based on multi-dimensional feature registration according to claim 2, wherein: The resampling method includes one of voxel gridding, nearest neighbor sampling, and surface-based sampling.

8. The method for fusing two-dimensional images and point clouds based on multi-dimensional feature registration according to claim 1, wherein: The pooling layer uses the maximum pooling method to select the maximum value of a certain area in the feature map as the downsampled feature.

9. The method for fusing two-dimensional images and point clouds based on multi-dimensional feature registration according to claim 1, wherein: The activation function is the softmax function, which generates the probability distribution of each category.

10. A device for the method for fusing two-dimensional images and point clouds based on multi-dimensional feature registration according to any one of claims 1 to 9, characterized in that: include: A preprocessing module, which is used to obtain a two-dimensional image of an object and corresponding three-dimensional point cloud data, and perform preprocessing on each of them; A registration and projection module is used to convert the pre-processed 2D image and 3D point cloud data into the same coordinate system, establish a spatial position association between the 2D image and the 3D point cloud; and project the registered 3D point cloud data into the corresponding 2D image space; A feature extraction module is used to extract features from two-dimensional images and three-dimensional point cloud data that have a spatial relationship with each other using a convolutional neural network to obtain corresponding feature representations; The feature fusion prediction module is used to build a deep learning model. The deep learning model includes an encoder and a decoder. The encoder is mainly composed of a convolution layer and a pooling layer. The convolution layer includes multiple convolution kernels of different scales. Each convolution kernel performs a convolution operation with the input data, retaining spatial structure information and extracting local features at the same time. The convolution kernel is the weight in the convolution operation. The convolution operation performs a sliding window calculation on the input data and generates a specific feature map. The feature map is activated by a nonlinear activation function to introduce nonlinear features; the pooling layer downsamples the feature map; After multiple convolutional and pooling layer processing, a fully connected layer is used to perform matrix multiplication and nonlinear transformation on the input shared feature representation and weights to generate output features; The decoder is mainly composed of dedicated convolutional layers and convolutional layers. The dedicated convolutional layers upsample the output features to increase the resolution, and then pass them through the convolutional layers for classification prediction. The output is consistent with the input image size and predicts the semantic category for each pixel. And for model training and optimization, the deep learning model constructed in step S6 is trained using the labeled training data set, and feature descent optimization is performed according to the loss function. After iterative training and verification, a prediction model is obtained; The output module is used to input the two-dimensional image of the object to be processed and the corresponding three-dimensional point cloud data after being processed in steps S1 to S5 into the prediction model in step S7 to obtain the fusion information of the object to be processed.

Images