A point cloud density enhancement method based on near-infrared image and millimeter wave radar point cloud fusion

Abstract

The application discloses a point cloud density enhancement method based on near-infrared image and millimeter wave radar point cloud fusion, and comprises the following steps: preprocessing original image data shot by a near-infrared camera; using a monocular depth estimation algorithm to perform absolute depth estimation on the preprocessed near-infrared image data, so as to obtain depth information of each pixel point of the preprocessed near-infrared image; converting the two-dimensional preprocessed near-infrared image data into three-dimensional near-infrared image point cloud data, and converting the three-dimensional near-infrared image point cloud data into a radar coordinate system; using a straight-through filter and ground point elimination algorithm to denoise the three-dimensional near-infrared image point cloud data in the radar coordinate system; using an ICP algorithm to register the original radar point cloud data and the denoised near-infrared image point cloud data, so as to obtain a corrected near-infrared image point cloud; using a K nearest neighbor algorithm to match the corrected near-infrared image with the original radar point cloud, and performing fusion, so as to enhance the density of the original radar point cloud. The application can work normally in all-weather conditions, such as extreme weather and night conditions.

Description

TECHNICAL FIELD

[0001] The present application relates to artificial intelligence depth estimation algorithm, monocular vision and millimeter wave radar sensor technology, and in particular to a point cloud enhancement method based on near-infrared image and millimeter wave radar point cloud fusion. BACKGROUND

[0002] With the rapid development of automatic driving, intelligent transportation and industrial automation, environmental perception technology has become a key link. As an important sensor, millimeter wave radar is widely used in vehicle detection, obstacle identification and distance measurement tasks due to its excellent performance in various harsh weather conditions such as rain, snow and fog. However, the point cloud data generated by millimeter wave radar is often sparse and has low resolution, making it difficult to provide detailed environmental details.

[0003] Near-infrared cameras have their unique advantages in environmental perception. Near-infrared light is not easily affected by visible light conditions and can still provide clear images in low-light environments, and has certain anti-dust and anti-fog capabilities. Combining near-infrared cameras with millimeter wave radars can make up for the limitations of single sensors and enhance the richness and accuracy of point cloud data.

[0004] The present application fuses millimeter wave radar point cloud data with near-infrared image data and designs a point cloud enhancement method based on near-infrared image and millimeter wave radar point cloud fusion. By fusing the perception data of near-infrared cameras and millimeter wave radars, the resolution and detail performance of point clouds can be significantly improved, making up for the sparsity of millimeter wave radar point clouds. It can provide more detailed spatial details for point cloud data, achieve more accurate environmental perception, and combine the reliability of millimeter wave radars in harsh weather conditions with the superior performance of near-infrared cameras in low-light environments to form an all-weather, all-environment perception system. This multi-sensor fusion technology can provide consistent and high-quality perception data under different environmental conditions, improving the robustness and reliability of the system. SUMMARY

[0005] The present application aims to provide a point cloud density enhancement method based on near-infrared image and millimeter wave radar point cloud fusion to solve the problem that the point cloud data generated by millimeter wave radars is usually sparse and has low resolution, making it difficult to provide detailed environmental details.

[0006] The solution to achieve the purpose of the present application is a point cloud density enhancement method based on near-infrared image and millimeter wave radar point cloud fusion, comprising the following steps:

[0007] Step 1, pre-process the original image data captured by the near-infrared camera, and the pre-processing operation includes image scaling and pixel data normalization operation;

[0008] Step 2, using monocular depth estimation algorithm to estimate the absolute depth of the pre-processed near-infrared image data, and obtain the depth information of each pixel point of the pre-processed near-infrared image;

[0009] Step 3, using the depth information of each pixel point of the pre-processed near-infrared image, converting the two-dimensional pre-processed near-infrared image data into three-dimensional near-infrared image point cloud data, and converting to the radar coordinate system;

[0010] Step 4, using straight-through filtering and ground point rejection algorithm to denoise the three-dimensional near-infrared image point cloud data in the radar coordinate system;

[0011] Step 5, using ICP algorithm to register the original radar point cloud data and the denoised near-infrared image point cloud data, and obtain the corrected near-infrared image point cloud;

[0012] Step 6, using K nearest neighbor algorithm to match the corrected near-infrared image and the original radar point cloud, and performing fusion to enhance the density of the original radar point cloud.

[0013] Further, step 1, pre-processing the original image data captured by the near-infrared camera, the pre-processing operation includes image scaling and pixel data normalization operation, the specific method is:

[0014] Suppose the original image resolution is WxH, and the size of the pre-processed image data is 1064x616. The original image data is scaled by the same ratio, and the scaling factor scale is calculated. The remaining pixels are filled with gray bars to make the resolution 1064x616. The scaling factor scale is calculated as follows:

[0015]

[0016] The normalized operation is performed on the scaled near-infrared image. According to the statistics, the mean value μ and the standard deviation σ of the pixel value are obtained. For each pixel value I c (i,j), where c represents the channel, i and j represent the row and column of the pixel respectively, and the normalized pixel value I' c (i,j) is represented as:

[0017]

[0018] Further, step 2, using monocular depth estimation algorithm to estimate the absolute depth of the pre-processed near-infrared image data, and obtain the depth information of each pixel point of the pre-processed near-infrared image, the specific method is:

[0019] (A) Network architecture design

[0020] The monocular depth estimation algorithm adopts a Vision Transformer structure to predict absolute depth, including an encoder module and a decoder module, the near-infrared image is input into the encoder for encoding, and four features of different sizes are extracted; in the decoder module, the four features of different sizes are taken as input for feature decoding, and the final depth map is output;

[0021] (c) Encoder module

[0022] The encoder module includes Patch Embedding, Position Embedding, Transformer Blocks, and Layer Nomalization submodules, which are executed in series;

[0023] The Patch Embedding module uses a 14x14 convolution with a step size of 14 to divide the input image into a plurality of non-overlapping patches, and embeds each patch into a vector;

[0024] The Position Embedding module adds position information to each patch embedding to preserve the positional relationship of the patches in the image, and adds a special marker symbol [CLS] for classification;

[0025] Twelve identical Transformer Block structures are connected in series, in the Transformer Block, the input is sequentially subjected to layer normalization 1, a multi-head self-attention mechanism layer, Layer Scale 1, layer normalization 2, a feedforward neural network, Layer Scale 2, the output of the multi-head self-attention mechanism layer and the output of Layer Scale 1 are connected in residual to serve as the input of layer normalization 2, the output of the feedforward neural network and the output of Layer Scale 2 are connected in residual to serve as the output of the Transformer Block;

[0026] The Layer Nomalization normalizes the output of the Transformer Blocks to obtain four features of different sizes;

[0027] (d) Decoder module

[0028] The decoder module comprises Token2feature, DecoderFeature, Depth Regressor and NormalPredictor, ContextFeatureEncoder, GRU Update Block sub-modules, and the output of the encoder sequentially passes through Token2feature and DecoderFeature sub-modules, then enters two branches, respectively passes through Depth Regressor and NormalPredictor sub-modules, and then is combined into ContextFeatureEncoder sub-module, and finally enters GRU Update Block sub-module to obtain the output;

[0029] Token2feature converts the output of the encoder into a multi-scale feature map, facilitating subsequent depth and normal prediction;

[0030] DecoderFeature decodes the multi-scale feature into initial depth and normal feature maps, comprising three FuseBlock sub-modules for feature fusion and up-sampling operations, and FusionBlock is composed of two 3x3 convolution layers with a step of 1 and an up-sampling layer, and through three times of feature fusion and up-sampling, a high-resolution depth and normal feature map is gradually generated;

[0031] Depth Regressor is used to convert the high-resolution depth feature map into a depth probability distribution and calculate the depth expectation value, and is composed of two 3x3 convolution layers with a step of 1 and a Relu activation function;

[0032] Normal Predictor is used to predict the normal from the high-resolution normal feature map, and is composed of two 3x3 convolution layers with a step of 1, four 1x1 convolution layers with a step of 1 and two Relu activation functions;

[0033] ContextFeatureEncoder encodes the outputs of Depth Regressor and Normal Predictor as input features to generate the initial hidden state and context features required by GRU Update, and ContextFeatureEncoder comprises ResidualBlock sub-module and convolution operation, which is responsible for converting the input features into context features, ResidualBlock comprises two 3x3 convolution layers with a step of 1, two normalization layers and two Relu activation functions, and the input of ResidualBlock is added to the output feature processed by ResidualBlock through residual connection operation to generate the output;

[0034] The GRU Update takes the output of the BlockContextFeatureEncoder as input to progressively optimize the depth and normal prediction, the GRU Update Block contains a ConvGRU and a FlowHead submodule, the ConvGRU contains three 3x3 convolutional layers with a stride of 1 (convz, convr, convq) to update the hidden state of the GRU, the FlowHead contains four 3x3 convolutional layers with a stride of 1 and an activation function to generate the incremental update of the depth and normal, the generated depth information is the required depth map.

[0035] Further, in step 3, the depth information of each pixel point of the preprocessed near-infrared image is used to convert the two-dimensional preprocessed near-infrared image data into three-dimensional near-infrared image point cloud data, and convert it to the radar coordinate system, the specific method is as follows:

[0036] Let the extrinsic parameter of the near-infrared camera to the millimeter wave radar be T, and the intrinsic parameter K of the near-infrared camera is as follows, where fx, fy are the focal lengths of the camera in the horizontal and vertical directions, and cx, cy are the principal point coordinates of the camera in the horizontal and vertical directions.

[0037]

[0038] Let any one pixel in the near-infrared image be (u, v) in the image plane, and its corresponding depth value be d(u, v), convert this pixel coordinate to the three-dimensional point coordinate (x, y, z) of the near-infrared image data in the camera coordinate system, and the following equation relationship is obtained:

[0039] d(u, v)·[u, v, 1] T = K·[x, y, z] T

[0040] Further derivation obtains the three-dimensional point coordinate calculation formula of the near-infrared image data as:

[0041]

[0042] Let a point P c in the camera coordinate system be converted to the radar coordinate system point P r , the method is as follows:

[0043] [P r , 1] T = [P c , 1] T · T

[0044] Through twice coordinate system transformation, the near-infrared image data is converted from the image plane coordinate system to the radar coordinate system, and the three-dimensional near-infrared image point cloud data is obtained by repeating the above operation on all pixel data on the whole near-infrared image.

[0045] Further, in step 4, the three-dimensional near-infrared image point cloud data in the radar coordinate system is denoised using a pass-through filtering and ground point elimination algorithm, and the specific method is as follows:

[0046] In step 4-1, the three-dimensional near-infrared image point cloud data in the radar coordinate system is denoised using a pass-through filtering, and points outside the effective distance and effective height of the near-infrared camera are filtered out. Assuming that the effective distance of the near-infrared camera is (DisL, DisH), the effective height of the detection is (HeightL, HeightH), and the pass-through filtering method is as follows:

[0047] Let the three-dimensional near-infrared image point cloud data set P in the radar coordinate system be {p i |1≤i≤N}, where i represents the i-th point, N represents the total number of points in the image point cloud, and the three-dimensional coordinates of each point can be represented as p i =(p ix ,p iy ,p iz ), and the image point cloud set after pass-through filtering is represented as:

[0048] P'={p i |1≤i≤N,DisL<p ix <DisH,HeightH<p iz <HeightH}

[0049] In step 4-2, the ground points are eliminated using a ground point elimination algorithm.

[0050] First, a pair of two-dimensional grid maps Map1 and Map2 are created to record the maximum and minimum point cloud heights in each grid, respectively.

[0051] Let P be the image point cloud set, where p i represents the i-th point, (x i ,y i ,z i ) is its coordinate, and the grid index idx of each point is:

[0052]

[0053] where Δx and Δy are the resolutions of the grid in the x and y directions, respectively, and x min and y minThe minimum coordinate of the grid map. Project each point in the image point cloud into the corresponding grid and update the height information in the grid map;

[0054]

[0055] For each grid, calculate the difference between the corresponding elements in the grid map recording the maximum height and the grid map recording the minimum height, to obtain the height difference grid map MapDiff:

[0056] MapDiff idx =Map1 idx -Map2 idx

[0057] Traverse each point and check if the height difference of the grid where the height difference grid map is located is greater than the threshold threshold. If it is greater than threshold, it is considered that the point in the grid is not a ground point, and it is retained in the non-ground point cloud set p'. If it is less than or equal to threshold, it is considered to be a ground point and it is removed. The method is as follows:

[0058] p' = {p | MapDiff idx > threshold}.

[0059] Further, step 5, use the ICP algorithm to register the radar original point cloud data and the denoised near-infrared image point cloud data, to obtain the corrected near-infrared image point cloud, the specific method is:

[0060] Let the radar original point cloud set The denoised near-infrared image point cloud set Where The ICP algorithm finds a transformation matrix T, so that p c After transformation and p r Align as much as possible;

[0061] The transformation matrix T is decomposed into a rotation matrix R and a translation vector t, that is:

[0062]

[0063] The goal of the ICP algorithm is to minimize the following error metric:

[0064]

[0065] Where Indicates the distance The nearest radar point, w i is the distance-based weight:

[0066]

[0067] wherein ε is a very small integer to avoid the situation of two points coinciding;

[0068] After multiple iterations, a transformation matrix is obtained, that is, a corrected camera-to-radar extrinsic matrix T', wherein the rotation matrix is R' and the translation vector is t'. The corrected extrinsic matrix is used to obtain a corrected near-infrared image point cloud The correction method is as follows:

[0069]

[0070] Further, in step 6, the K-Nearest Neighbor algorithm is used to match the corrected near-infrared image and the original radar point cloud, and fusion is performed to enhance the density of the original radar point cloud. The specific method is as follows:

[0071] Let the original radar point cloud be P The K-Nearest Neighbor algorithm is used to find the K nearest neighbor image points of P in the corrected near-infrared image point cloud P The corresponding Euclidean distance is d The image point weight is represented as:

[0072]

[0073] wherein σ is a decay factor, the maximum image point confidence is 0.5, and the radar point weight is w r =1-w c The fused point cloud set is represented as P The fusion formula is:

[0074]

[0075] The enhanced point cloud set P enhance is the superposition of the original radar point cloud set and the fused point cloud set, and is represented as: P enhance =M∩P r

[0076] A point cloud enhancement system based on near-infrared image and millimeter wave radar point cloud fusion, implements the point cloud enhancement method based on near-infrared image and millimeter wave radar point cloud fusion, and realizes point cloud enhancement based on near-infrared image and millimeter wave radar point cloud fusion.

[0077] A computer device includes a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the computer program, the point cloud enhancement method based on near-infrared image and millimeter wave radar point cloud fusion is implemented, and point cloud enhancement based on near-infrared image and millimeter wave radar point cloud fusion is realized.

[0078] A computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, implements the point cloud enhancement method based on fusion of near-infrared images and millimeter wave radar point clouds, realizes point cloud enhancement based on fusion of near-infrared images and millimeter wave radar point clouds.

[0079] Compared with the prior art, the present application has the following advantages: (1) By fusing high-resolution image data of a near-infrared camera, the resolution and detail performance of the millimeter wave radar point cloud are significantly improved. The near-infrared image provides rich detail information, which can effectively supplement the sparseness of the millimeter wave radar point cloud, making the overall point cloud data more fine and accurate. (2) The combination of the near-infrared camera and the millimeter wave radar enables the perception system to provide consistent and high-quality perception data in various environmental conditions, such as severe weather, low-light environment and complex lighting conditions. The millimeter wave radar performs well in severe weather conditions such as rain, snow and fog, while the near-infrared camera has excellent performance in night or low-light environment. The combination of the two can form an all-weather and all-environment perception system. BRIEF DESCRIPTION OF DRAWINGS

[0080] Figure 1 The algorithm framework of the present application is shown in the figure.

[0081] Figure 2 The absolute depth estimation encoder framework is shown in the figure.

[0082] Figure 3 The absolute depth estimation decoder framework is shown in the figure.

[0083] Figure 4 The depth estimation and point cloud density enhancement effect diagram is shown in the figure.

[0084] Figure 5 The model target detection comparison effect diagram before and after point cloud enhancement is shown in the figure. DETAILED DESCRIPTION

[0085] In order to make the purpose, technical scheme and advantages of the present application more clear, the present application is further described in detail below in combination with the drawings and examples. It should be understood that the specific examples described herein are only used to explain the present application and do not limit the present application.

[0086] As shown in the figure, the present application is a point cloud enhancement method based on fusion of near-infrared images and millimeter wave radar point clouds, which includes the following steps: Figure 1 Step 1, pre-processing the original image data taken by the near-infrared camera, the pre-processing operation includes image scaling and pixel data normalization operation;

[0087]

[0088] ​Let the original image resolution be W x H, and the size of the image data after preprocessing be 1064 x 616. The original image data is scaled by the same ratio, and the scaling factor scale is calculated. The remaining pixels are filled with gray bars to make the resolution 1064 x 616. The scaling factor scale is calculated as follows:

[0089]

[0090] The scaled near-infrared image is normalized. According to the statistics, the mean μ and standard deviation σ of the pixel value are obtained. For each pixel value I c (i,j), where c represents the channel, i and j represent the row and column of the pixel respectively, and the normalized pixel value I' c (i,j) is represented as:

[0091]

[0092] Step 2, use monocular depth estimation algorithm to estimate the absolute depth of the preprocessed near-infrared image data, and get the depth information of each pixel point of the preprocessed near-infrared image;

[0093] (A) Network architecture design

[0094] The monocular depth estimation algorithm uses the Vision Transformer structure to predict the absolute depth, including an encoder module and a decoder module. The near-infrared image is input into the encoder for encoding, and four different size features are extracted; in the decoder module, the four different size features are input as input, and the final depth map is output after decoding;

[0095] (e) Encoder module

[0096] The encoder module includes Patch Embedding, Position Embedding, Transformer Blocks, and Layer Nomalization submodules, which are executed in series.

[0097] The Patch Embedding module uses a 14 x 14 convolution with a step size of 14 to divide the input image into several non-overlapping patches, and each patch is embedded into a vector.

[0098] The Position Embedding module adds position information to each patch embedding to preserve the position relationship of the patches in the image, and adds a special marker symbol [CLS] for classification.

[0099] The 12 same Transformer Block structures are connected in series, in the Transformer Block, the input is sequentially subjected to layer normalization 1, a multi-head self-attention mechanism layer, Layer Scale 1, layer normalization 2, a feedforward neural network, Layer Scale 2, wherein the output of the multi-head self-attention mechanism layer and the output of Layer Scale 1 are connected in residual to serve as the input of layer normalization 2, the output of the feedforward neural network and the output of Layer Scale 2 are connected in residual to serve as the output of the Transformer Block;

[0100] The Layer Nomalization normalizes the output of the Transformer Blocks to obtain four different sizes of features;

[0101] (f) Decoder module

[0102] The decoder module includes Token2feature, DecoderFeature, Depth Regressor and NormalPredictor, ContextFeatureEncoder, GRU Update Block sub-modules, the output of the encoder is sequentially subjected to Token2feature and DecoderFeature sub-modules, then enters two branches, respectively subjected to Depth Regressor and Normal Predictor sub-modules, and then combined into ContextFeatureEncoder sub-module, and finally enters GRU Update Block sub-module to obtain the output;

[0103] Token2feature converts the output of the encoder into a multi-scale feature map, which is convenient for subsequent depth and normal prediction;

[0104] DecoderFeature decodes the multi-scale features into initial depth and normal feature maps, including three FuseBlock sub-modules for feature fusion and upsampling operations, FusionBlock is composed of two 3*3 convolution layers with a step of 1 and an upsampling layer, and through three times of feature fusion and upsampling, high-resolution depth and normal feature maps are gradually generated;

[0105] Depth Regressor is used to convert the high-resolution depth feature map into a depth probability distribution and calculate the depth expectation value, which is composed of two 3*3 convolution layers with a step of 1 and a Relu activation function;

[0106] The normal predictor is used to predict the normal from the high-resolution normal feature map, which is composed of two 3x3 stride 1 convolution layers, four 1x1 stride 1 convolution layers and two Relu activation functions;

[0107] The context feature encoder encodes the output of the depth regressor and the normal predictor as input features to generate the initial hidden state and context features required for the GRU update, and the context feature encoder includes a residual block submodule and a convolution operation, which is responsible for converting the input features into context features, the residual block includes two 3x3 stride 1 convolution layers, two normalization layers and two Relu activation functions, and the input of the residual block is added to the output feature processed by the residual block through a residual connection operation to generate the output;

[0108] The GRU update takes the output of the block context feature encoder as input to gradually optimize the depth and normal prediction, and the GRU update block includes a ConvGRU and a FlowHead submodule, the ConvGRU includes three 3x3 stride 1 convolution layers (convz, convr, convq) for updating the hidden state of the GRU, and the FlowHead includes four 3x3 stride 1 convolution layers and an activation function for generating incremental updates of the depth and normal, and the generated depth information is the required depth map.

[0109] Step 3, using the depth information of each pixel point of the preprocessed near-infrared image, convert the two-dimensional preprocessed near-infrared image data into three-dimensional near-infrared image point cloud data, and convert to the radar coordinate system;

[0110] Let the extrinsic parameter of the near-infrared camera to the millimeter wave radar be T, and the intrinsic parameter K of the near-infrared camera is as follows, where fx and fy are the focal lengths of the camera in the horizontal and vertical directions, and cx and cy are the principal point coordinates of the camera in the horizontal and vertical directions.

[0111]

[0112] Let the coordinates of any one pixel in the near-infrared image be (u, v), and the corresponding depth value be d(u, v), and convert the pixel coordinates into the three-dimensional point coordinates (x, y, z) of the near-infrared image data in the camera coordinate system, which has the following equation relationship:

[0113] d(u, v) · [u, v, 1] T = K · [x, y, z] T

[0114] Further derivation of the three-dimensional point coordinate calculation formula of near-infrared image data is as follows:

[0115]

[0116] Suppose that a point P in the camera coordinate system is c converted to the radar coordinate system point P r The method is as follows:

[0117] [P r ,1] T = [P c ,1] T · T

[0118] Through twice coordinate system transformation, the near-infrared image data is converted from the image plane coordinate system to the radar coordinate system, and the three-dimensional near-infrared image point cloud data is obtained by repeating the above operation on all pixel data on the entire near-infrared image.

[0119] Step 4, using a pass-through filter and ground point rejection algorithm to denoise the three-dimensional near-infrared image point cloud data in the radar coordinate system;

[0120] Step 4-1, using a pass-through filter to denoise the three-dimensional near-infrared image point cloud data in the radar coordinate system, and filtering out points outside the effective distance and effective height of the near-infrared camera, assuming that the effective distance of the near-infrared camera is (DisL, DisH), and the effective height of the detection is (HeightL, HeightH), the pass-through filter method is as follows:

[0121] Suppose that the three-dimensional near-infrared image point cloud data set in the radar coordinate system is P = {p i |1≤i≤N}, where i represents the i-th point, N represents the total number of points in the image point cloud, and the three-dimensional coordinates of each point can be represented as p i = (p ix , p iy , p iz ), and the image point cloud set after pass-through filtering is represented as:

[0122] P' = {p i |1≤i≤N, DisL < p ix < DisH, HeightH < p iz < HeightH}

[0123] Step 4-2, using a ground point rejection algorithm to reject invalid ground points;

[0124] First, create a pair of two-dimensional grid maps Map1 and Map2 to record the maximum and minimum values of the point cloud height in each grid, respectively;

[0125] Let be the image point cloud set, where p i represents the i-th point, (x i , y i , z i ) is its coordinate, and the grid index idx of each point is:

[0126]

[0127] where Δx and Δy are the resolutions of the grid in the x and y directions, respectively, and x min and y min are the minimum coordinates of the grid map. Project each point in the image point cloud into the corresponding grid and update the height information in the grid map;

[0128]

[0129] For each grid, calculate the difference between the corresponding elements in the grid map recording the maximum height and the grid map recording the minimum height to obtain the height difference grid map MapDiff:

[0130] MapDiff idx = Map1 idx - Map2 idx

[0131] Iterate through each point and check if the height difference of the grid where its height difference grid map is greater than the threshold threshold. If it is greater than threshold, it is considered that the point in the grid is not a ground point, and it is retained in the non-ground point cloud set p'. If it is less than or equal to threshold, it is considered to be a ground point and it is removed. The method is as follows:

[0132] p' = {p | MapDiff idx > threshold}.

[0133] Step 5, use the ICP algorithm to register the radar original point cloud data and the denoised near-infrared image point cloud data to obtain the corrected near-infrared image point cloud;

[0134] Let the radar original point cloud set be the denoised near-infrared image point cloud set where ICP algorithm finds a transformation matrix T, so that p c is aligned with p r as much as possible after transformation;

[0135] The transformation matrix T is decomposed into a rotation matrix R and a translation vector t, that is:

[0136]

[0137] The objective of the ICP algorithm is to minimize the following error metric:

[0138]

[0139] Where p r i Represents distance p c i The nearest radar point, w i For distance-based weights:

[0140]

[0141] Here, ε is a very small integer to avoid the situation where two points coincide;

[0142] After multiple iterations, a transformation matrix is ​​obtained, namely the corrected camera-to-radar extrinsic parameter matrix T′, where the rotation matrix is ​​R′ and the translation vector is t′. The corrected near-infrared image point cloud is obtained using the corrected extrinsic parameter matrix. The correction method is as follows:

[0143]

[0144] Step 6: Use the K-nearest neighbor algorithm to match and fuse the corrected near-infrared image with the original radar point cloud to enhance the density of the original radar point cloud.

[0145] Let the original radar point cloud be... The K-nearest neighbor algorithm was used to find the point clouds of the corrected near-infrared image. The K nearest neighbor image points Its corresponding Euclidean distance Image point weights are represented as follows:

[0146]

[0147] Where σ is the attenuation factor, the maximum confidence level of the image point is 0.5, and the radar point weight is w. r =1-w c The fused point cloud set is represented as The fusion formula is:

[0148]

[0149] Enhanced point cloud set P enhanceFor the superposition of the original radar point cloud set and the fused point cloud set, it is expressed as:

[0150] P enhance = M \ P r

[0151] The application further provides a point cloud enhancement system based on near-infrared image and millimeter wave radar point cloud fusion, and the point cloud enhancement method based on near-infrared image and millimeter wave radar point cloud fusion is implemented to realize point cloud enhancement based on near-infrared image and millimeter wave radar point cloud fusion.

[0152] A computer device comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and when the processor executes the computer program, the point cloud enhancement method based on near-infrared image and millimeter wave radar point cloud fusion is implemented to realize point cloud enhancement based on near-infrared image and millimeter wave radar point cloud fusion.

[0153] A computer readable storage medium has a computer program stored thereon, and when the computer program is executed by a processor, the point cloud enhancement method based on near-infrared image and millimeter wave radar point cloud fusion is implemented to realize point cloud enhancement based on near-infrared image and millimeter wave radar point cloud fusion.

[0154] Embodiment

[0155] In order to verify the effectiveness of the application scheme, the following experiment is carried out.

[0156] The mainstream point cloud based target detection model PointPillar is selected, and the model is trained on the original millimeter wave radar point cloud and the point cloud with enhanced point cloud density by the application respectively, and the effect in the daytime and night scene is detected. The depth estimation effect of the near-infrared image and the enhancement effect of the point cloud density enhancement method in the night condition are as shown in Figure 4 , wherein Figure 4 -a, the original image of the near-infrared image, Figure 4 -b, 4-c respectively represent the image point cloud color drawing and the image point cloud, Figure 4 -e, 4-f respectively represent the original millimeter wave radar point cloud and the enhanced point cloud after the point cloud density enhancement. The performance of PointPillar on the original point cloud and the enhanced point cloud is as shown in Figure 5 , wherein Figure 5 -a, 5-b is the night scene, and 5-c, 5-d is the daytime scene. It can be seen that the false detection rate and the missed detection rate of the PointPillar model trained on the enhanced point cloud are greatly reduced, and the accuracy and robustness are better.

[0157] The above-described embodiments are merely illustrative of several embodiments of the present application, which are described in more detail and in a specific manner, but should not be construed as limiting the scope of the present application. It should be noted that, for those skilled in the art, several modifications and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the appended claims.

Claims

1. A point cloud density enhancement method based on near-infrared image and millimeter wave radar point cloud fusion, characterized by, The method comprises the following steps: Step 1, preprocessing the original image data taken by the near-infrared camera, the preprocessing operation including image scaling and pixel data normalization operation; Step 2, using monocular depth estimation algorithm to perform absolute depth estimation on the preprocessed near-infrared image data, obtaining the depth information of each pixel point of the preprocessed near-infrared image; Step 3, using the depth information of each pixel point of the preprocessed near-infrared image, converting the two-dimensional preprocessed near-infrared image data into three-dimensional near-infrared image point cloud data, and converting to the radar coordinate system; Step 4, using straight-through filtering and ground point rejection algorithm to denoise the three-dimensional near-infrared image point cloud data in the radar coordinate system; Step 5, using ICP algorithm to register the original radar point cloud data and the denoised near-infrared image point cloud data, obtaining the corrected near-infrared image point cloud; Step 6, using K nearest neighbor algorithm to match the corrected near-infrared image and the original radar point cloud and perform fusion to enhance the density of the original radar point cloud, the specific method being: Let the radar raw point cloud be The K nearest neighbors of each point in the corrected near-infrared image point cloud are found using the K-Nearest Neighbors algorithm Their corresponding Euclidean distances are The image point weight is represented as: where σ is an attenuation factor, the image point confidence is maximally 0.5, and the radar point weight is w r = 1 - w c The fused point cloud set is denoted as The fusion formula is: Enhanced point cloud set P enhance For the overlay of the original radar point cloud set and the fused point cloud set, denoted as: P enhance = M ∩ P r .

2. The point cloud density enhancement method based on near-infrared image and millimeter wave radar point cloud fusion according to claim 1, characterized in that, Step 1, preprocessing the original image data taken by the near-infrared camera, the preprocessing operation including image scaling and pixel data normalization operation, the specific method being: Let the original image resolution be WxH, and the size of the image data after preprocessing be 1064x616, scale the original image data by the same ratio, calculate the scaling factor scale, and fill the remaining pixels with gray bars to make the resolution 1064x616, the calculation method of the scaling factor scale being as follows: The normalized near-infrared image is subjected to a normalization operation, and the mean μ and standard deviation σ of the pixel values are obtained according to statistics. For each pixel value I c (i,j), where c represents a channel, i and j represent the row and column of the pixel, respectively, and the normalized pixel value I' c (i,j) is represented as:

3. The method of claim 1, wherein the method is based on fusing near-infrared images and millimeter wave radar point clouds. Step 2, using monocular depth estimation algorithm to perform absolute depth estimation on the preprocessed near-infrared image data, obtaining the depth information of each pixel point of the preprocessed near-infrared image, the specific method being: (A) Network architecture design The monocular depth estimation algorithm uses the Vision Transformer structure to predict the absolute depth, including an encoder module and a decoder module, the near-infrared image is input into the encoder for encoding, and four different size features are extracted; In the decoder module, the four different size features are used as input for feature decoding, and the final depth map is output; (a) Encoder module The encoder module includes Patch Embedding, Position Embedding, Transformer Blocks, and LayerNomalization submodules, which are executed in series; The Patch Embedding module uses a 14x14 convolution with a step size of 14 to divide the input image into a plurality of non-overlapping patches, and embeds each patch into a vector; The Position Embedding module adds position information to each patch embedding to preserve the positional relationship of the patches in the image, and adds a special marker symbol [CLS] for classification; The 12 same Transformer Block structures are connected in series, in the Transformer Block, the input is sequentially subjected to layer normalization 1, a multi-head self-attention mechanism layer, Layer Scale 1, layer normalization 2, a feedforward neural network, Layer Scale 2, wherein the output of the multi-head self-attention mechanism layer and the output of Layer Scale 1 are connected in residual connection as the input of layer normalization 2, the output of the feedforward neural network and the output of Layer Scale 2 are connected in residual connection as the output of the Transformer Block; Layer Nomalization normalizes the output of the Transformer Blocks to obtain four different sizes of features; (b) a decoder module The decoder module comprises Token2feature, DecoderFeature, Depth Regressor and NormalPredictor, ContextFeatureEncoder, GRU Update Block submodules, the output of the encoder is sequentially subjected to Token2feature and DecoderFeature submodules, then enters two branches, is subjected to Depth Regressor and Normal Predictor submodules respectively, is combined into the ContextFeatureEncoder submodule, and finally enters the GRU Update Block submodule to obtain the output; Token2feature converts the output of the encoder into a multi-scale feature map, which is convenient for subsequent depth and normal prediction; DecoderFeature decodes the multi-scale features into initial depth and normal feature maps, comprising three FuseBlock submodules for feature fusion and upsampling operations, the FusionBlock is composed of two 3*3 convolution layers with a step of 1 and an upsampling layer, and high-resolution depth and normal feature maps are gradually generated through three times of feature fusion and upsampling; Depth Regressor is used for converting the high-resolution depth feature map into a depth probability distribution and calculating a depth expectation value, and is composed of two 3*3 convolution layers with a step of 1 and a Relu activation function; Normal Predictor is used for predicting a normal from a high-resolution normal feature map, and is composed of two 3*3 convolution layers with a step of 1, four 1*1 convolution layers with a step of 1 and two Relu activation functions; The ContextFeatureEncoder encodes the outputs of the Depth Regressor and the Normal Predictor as input features to generate the initial hidden state and context features required for the GRU Update, the ContextFeatureEncoder includes a ResidualBlock submodule and a convolution operation, which is responsible for converting the input features into context features, the ResidualBlock includes two 3x3 convolution layers with a step size of 1, two normalization layers, and two Relu activation functions, the input of the ResidualBlock is added to the output feature processed by the ResidualBlock through a residual connection operation to generate the output; The GRU Update takes the output of the BlockContextFeatureEncoder as input to gradually optimize the depth and normal prediction, the GRU Update Block includes a ConvGRU and a FlowHead submodule, the ConvGRU includes three 3x3 convolution layers (convz, convr, convq) with a step size of 1 for updating the hidden state of the GRU, and the FlowHead includes four 3x3 convolution layers with a step size of 1 and an activation function for generating incremental updates of the depth and normal, and the generated depth information is the required depth map.

4. The method of claim 1, wherein the method is based on fusing near-infrared images and millimeter wave radar point clouds. Step 3, using the depth information of each pixel point of the preprocessed near-infrared image, convert the two-dimensional preprocessed near-infrared image data into three-dimensional near-infrared image point cloud data, and convert it to the radar coordinate system, the specific method is as follows: Let the extrinsic parameter of the near-infrared camera to the millimeter wave radar be T, and the intrinsic parameter K of the near-infrared camera be as follows, where fx and fy are the focal lengths of the camera in the horizontal and vertical directions, and cx and cy are the principal point coordinates of the camera in the horizontal and vertical directions; Let any one pixel in the near-infrared image be (u, v) in the image plane, and its corresponding depth value be d(u, v), convert this pixel coordinate to the three-dimensional point coordinate (x, y, z) of the near-infrared image data in the camera coordinate system, which has the following equation relationship: d(u,v) · [u,v, 1] T = K · [x,y,z] T Further derivation gives the calculation formula of the three-dimensional point coordinates of the near-infrared image data as: Let P be a point in the camera coordinate system c Transforming to the radar coordinate system point P r The method is as follows: [P r ,1] T = [P c ,1] T · T Through two coordinate system transformations, the near-infrared image data is converted from the image plane coordinate system to the radar coordinate system, and the above operation is repeated for all pixel data on the entire near-infrared image to obtain three-dimensional near-infrared image point cloud data.

5. The method of claim 1, wherein, Step 4, use the straight-through filtering and ground point rejection algorithm to denoise the three-dimensional near-infrared image point cloud data in the radar coordinate system, the specific method is as follows: Step 4-1, use straight-through filtering to denoise the three-dimensional near-infrared image point cloud data in the radar coordinate system, filter out points outside the effective distance and effective height of the near-infrared camera, let the effective distance of the near-infrared camera be (DisL, DisH), and the effective height of the detection be (HeightL, HeightH), the straight-through filtering method is as follows: A three-dimensional near-infrared image point cloud data set P = {p i |1≤i≤N} in a radar coordinate system is provided, where i represents the i-th point, N represents the total number of points in the image point cloud, and the three-dimensional coordinates of each point can be represented as p i =(p ix ,p iy ,p iz ), and the image point cloud set after the straight-through filtering is represented as: P' = {p i |1≤i≤N, DisL < p ix < DisH, HeightH < p iz < HeightH} Step 4-2, using the ground point removal algorithm, remove invalid ground points; First, create a pair of two-dimensional grid maps Map1 and Map2 to record the maximum and minimum values of the point cloud height in each grid, respectively; Set of image point clouds, where p i represents the i-th point, (x i ,y i ,z i ) are its coordinates, and the grid index idx of each point is: where Δx and Δy are the resolutions of the grid in x and y directions, respectively, x min and y min are the minimum coordinates of the grid map, and each point in the image point cloud is projected into the corresponding grid, and the height information in the grid map is updated. For each grid, calculate the difference between the corresponding elements in the grid map recording the maximum height and the grid map recording the minimum height to obtain the height difference grid map MapDiff: MapDiff idx = Map1 idx - Map2 idx Traverse each point and check whether the height difference of the grid where the height difference grid map is located is greater than the threshold threshold. If it is greater than threshold, it is considered that the point in the grid is not a ground point, and it is retained in the non-ground point cloud set p'. If it is less than or equal to threshold, it is considered to be a ground point, and it is removed. The method is as follows: p' = {p | MapDiff idx > threshold}.

6. The method of claim 1, wherein the method is based on fusing near-infrared images with millimeter wave radar point clouds. Step 5, use the ICP algorithm to register the radar original point cloud data and the denoised near-infrared image point cloud data to obtain the corrected near-infrared image point cloud. The specific method is as follows: Set of radar raw point clouds Set of denoised near-infrared image point clouds wherein The ICP algorithm finds a transformation matrix T such that p c After transformation and p r are aligned as much as possible; The transformation matrix T is decomposed into a rotation matrix R and a translation vector t, that is: The goal of the ICP algorithm is to minimize the following error metric: wherein represents the distance the closest radar point, w i is a distance-based weight: Where ε is a very small integer to avoid the case where two points coincide. After multiple iterations, a transformation matrix is obtained, that is, a corrected camera-to-radar extrinsic matrix T', in which the rotation matrix is R' and the translation vector is t'. A corrected near-infrared image point cloud is obtained by using the corrected extrinsic matrix The correction method is as follows:

7. A point cloud enhancement system based on near-infrared image and millimeter wave radar point cloud fusion, characterized in that, Implementing the point cloud enhancement method based on the fusion of near-infrared images and millimeter wave radar point clouds according to any one of claims 1-6 to achieve point cloud enhancement based on the fusion of near-infrared images and millimeter wave radar point clouds.

8. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein when the processor executes the computer program, the point cloud enhancement method based on the fusion of near-infrared images and millimeter wave radar point clouds according to any one of claims 1-6 is implemented to achieve point cloud enhancement based on the fusion of near-infrared images and millimeter wave radar point clouds.

9. A computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the point cloud enhancement method based on the fusion of near-infrared images and millimeter wave radar point clouds according to any one of claims 1-6 is implemented to achieve point cloud enhancement based on the fusion of near-infrared images and millimeter wave radar point clouds.

Images