Sky-time scene adaptive three-dimensional target detection method based on radar and visual sensor fusion

Abstract

The application provides a scene self-adaptive three-dimensional target detection method based on radar and visual sensing fusion. First, original point cloud features and image features are extracted from input point cloud data and image data respectively. Then, the original point cloud is projected onto the image feature map through matrix transformation, and the point cloud projection points are fused point by point to obtain the point cloud fusion features. Then, a scene-driven three-dimensional target detection network is constructed. According to the weather time condition and target size in the current scene, a time-sensitive factor and a scene scale factor are proposed. According to the time-sensitive factor and the scene scale factor, the global feature fusion of the point-by-point fusion features and the original point cloud features is adaptively realized. The global fusion features are input into a three-dimensional region generation network to generate a region of interest. After the region features are extracted, they are input into the three-dimensional target detection network to complete the three-dimensional target detection. Finally, the trained three-dimensional target detection network is used for real-time detection of the target.

Description

TECHNICAL FIELD

[0001] The present application relates to a scene adaptive three-dimensional target detection method based on radar and vision sensor fusion, and belongs to the field of deep learning. BACKGROUND

[0002] In recent years, with the rapid development of deep learning and the emergence of high-performance graphics cards, the development of three-dimensional target detection methods has been greatly promoted. Three-dimensional target detection based on multi-source data can automatically fuse target features and complement information from different modalities, greatly improving detection accuracy and scene applicability. Therefore, three-dimensional target detection methods based on deep learning have sparked a research boom.

[0003] The problem to be solved by the three-dimensional target detection method based on multi-source data is to find a more accurate and effective method to utilize multi-source data to complement their inherent defects and improve the accuracy of the method. Most existing fusion methods directly project point clouds onto a two-dimensional plane, then extract two-dimensional plane features of point clouds and image features for simple splicing. However, this method has the problem of inconsistent domain distribution of different modalities, information redundancy in direct matching, and many interference factors. At the same time, in scenes with severe occlusion or low brightness, there are more feature noises, which may introduce harmful information in the feature fusion process, which will lead to a decrease in the accuracy of the detection result. SUMMARY

[0004] In order to overcome the shortcomings of the existing method, the present application provides a scene adaptive three-dimensional target detection method based on radar and vision sensor fusion. First, the original point cloud features and image features are extracted from the input point cloud data and image data respectively. Then, the original point cloud is projected onto the image feature map through matrix transformation, and the point-by-point feature fusion of the point cloud projection points is performed to obtain the point-by-point fusion features of the point cloud. Then, a scene-driven three-dimensional target detection network is constructed, and the weather time condition and target size in the current scene are used to propose the time and scene scale factors. The global feature fusion of the point-by-point fusion features and the original point cloud features is realized adaptively according to the time and scene scale factors, and the global fusion features are input into the three-dimensional region generation network to generate the region of interest, and the region features are extracted and sent into the three-dimensional target detection network to complete the three-dimensional target detection. Finally, the trained three-dimensional target detection network is used for real-time detection of the target.

[0005] To solve the above technical problems, the technical scheme adopted by the present application is as follows:

[0006] A scene adaptive three-dimensional target detection method based on radar and vision sensor fusion, the specific process is as follows:

[0007] Step 1, extract the original point cloud features and image features from the input point cloud data and image data respectively;

[0008] Step 2: Project the input point cloud data onto image features, and perform point-by-point feature fusion on the projected point cloud points to obtain the point-by-point fused features of the point cloud data.

[0009] Step 3: Construct a scene-driven 3D target detection network and use the trained 3D target detection network to detect targets in real time.

[0010] Furthermore, the specific process of extracting original point cloud features from the input point cloud data in step 1 is as follows:

[0011] For point cloud data, firstly, a point cloud data is randomly selected as the initial point cloud data; then, random sampling is performed on the point cloud data in sequence using the farthest point sampling method. After the sampling is completed, a sphere is formed with each sampled point cloud data as the center and a specified radius. All point cloud data contained in each sphere is called a cluster; finally, features are extracted from each point cloud data in each cluster and max pooling is performed in the channel dimension to obtain cluster features. All cluster features are combined to obtain the original point cloud features.

[0012] Furthermore, the specific process of extracting image features from the input image data in step 1 is as follows:

[0013] For image data, four concatenated convolutional blocks are first input, with each convolutional block followed by a batch normalization layer and a ReLU activation function; then, the output of the last convolutional block is subjected to four transposed convolutions of different lengths in parallel to obtain four feature maps of the same resolution. The four feature maps of the same resolution are then concatenated to obtain the image features.

[0014] Furthermore, the specific process of step 2 is as follows:

[0015] Step 2.1: Project the input point cloud data onto the image features;

[0016] Step 2.2, divide the point cloud projection points into point-by-point fusion regions:

[0017]

[0018] Among them, h i and w i Take the i-th point cloud data P as an example. i Point cloud projection point P on image features i The height and width of the point-by-point merging region when ′ is the center point; It is the floor function; φ i For P i The corresponding weight coefficient, For P i The corresponding discrete coefficients, and respectively represent the minimum value and the maximum value in the discrete coefficient set ; H and W respectively represent the height and the width of the image data; i, j ∈ [1, 2, …, L], L is the number of point cloud projection points; d ij is the distance between the point cloud projection point p i ' and the jth point cloud data P j on the image feature; j

[0019] Step 2.3, the point-by-point fusion area is evenly divided into I rows and J columns, and the coordinates of the center points of each grid are calculated;

[0020] Step 2.4, for the center point of each grid, the result of the bilinear interpolation and the result of the nearest neighbor interpolation are obtained;

[0021] Step 2.5, the bilinear interpolation result and the nearest neighbor interpolation result of the center point of each grid are weighted fused to obtain the interpolation feature vector of the corresponding grid;

[0022] Step 2.6, the center distance measurement coefficient corresponding to each grid is calculated and normalized;

[0023] Step 2.7, according to the normalized center distance measurement coefficient, the interpolation feature vectors of each grid are weighted summed to obtain the point fusion feature of the corresponding point cloud projection point;

[0024] Step 2.8, the point fusion features of all point cloud projection points are spliced to obtain the point-by-point fusion feature of the point cloud data.

[0025] Further, in the step 2.1, the input point cloud data is projected onto the image feature according to the following formula:

[0026] P i ' = K r [R c |T c ]P i

[0027] Where P i ' is the point cloud projection point of the ith point cloud data P i on the image feature; K r is the camera intrinsic matrix; R c is the rotation matrix, and T c is the projection matrix.

[0028] Further, in the step 2.3, the calculation formula of the coordinates of the grid center points is as follows:

[0029] ​

[0030] where (u ab ,v ab ) is the coordinate of the center point of the grid in the ath row and bth column in the point-by-point fusion region; a = 1, 2,... I, b = 1, 2,... J; h is the height of the point-by-point fusion region, w is the width of the point-by-point fusion region; (u, v) is the coordinate of the point cloud projection point in the point-by-point fusion region.

[0031] Further, in step 2.4, the calculation process of the bilinear interpolation result f ab_1 and the nearest neighbor interpolation result f ab_2 of the center point of the grid in the ath row and bth column in the point-by-point fusion region is as follows:

[0032] f ab_1 = f1(u-u1)(v-v1) + f2(u-u1)(v2-v) + f3(u2-u)(v-v1) + f4(u2-u)(v2-v)

[0033]

[0034] where (u1, v1), (u1, v2), (u2, v1), (u2, v2) are the coordinates of the four nearest neighbor point cloud projection points of the center point of the grid, respectively; d1, d2, d3, d4 are the distances from the four nearest neighbor point cloud projection points to the center point of the grid; f1, f2, f3, f4 are the feature vectors corresponding to the four nearest neighbor point cloud projection points, is the feature vector corresponding to the nearest neighbor point cloud projection point of the center point of the grid.

[0035] Further, in step 2.5, the calculation formula of the interpolation feature vector f ab of the grid in the ath row and bth column in the point-by-point fusion region is as follows:

[0036]

[0037] where f ab_1 and f ab_2 are the bilinear interpolation result and the nearest neighbor interpolation result of the center point of the grid in the ath row and bth column in the point-by-point fusion region, respectively; is the aggregation coefficient, N d is the length of the feature vector dimension, f ω_ψ represents the feature value of the ωth feature vector in the ψth dimension; μ ψ is the mean value of the value in the ψth dimension; σ ψ is the standard deviation of the value in the ψth dimension.

[0038] Further, in step 2.6, the center distance measurement coefficient η of the grid corresponding to the a-th row and the b-th column in the point-by-point fusion region is calculated as follows ab and the normalized result η ab is calculated as follows:

[0039]

[0040] where (u ab ,v ab ) is the coordinate of the center point of the grid corresponding to the a-th row and the b-th column in the point-by-point fusion region; min(u,u ab ) represents the smaller value of u ab and u; max(u,u ab ) represents the larger value of u ab and u; min(v,v ab ) represents the smaller value of v and v ab ; max(v,v ab ) represents the larger value of v and v ab ; and (u,v) is the coordinate of the point cloud projection point in the point-by-point fusion region.

[0041] Further, in step 2.7, the point fusion feature f LI of the point cloud projection point corresponding to the grid of the a-th row and the b-th column in the point-by-point fusion region is calculated as follows:

[0042]

[0043] where η ab ′ is the normalized result of the center distance measurement coefficient of the grid corresponding to the a-th row and the b-th column in the point-by-point fusion region, and f ab is the interpolation feature vector of the grid of the a-th row and the b-th column in the point-by-point fusion region.

[0044] Further, in step 2, before projecting the input point cloud data onto the image feature, the input point cloud data is further filtered.

[0045] Further, the specific process of step 3 is as follows:

[0046] Step 3.1, input the point-by-point fusion feature F LI and the original point cloud feature F L into the global feature fusion network for global feature fusion:

[0047] First, map F LI and F L to the same channel dimension through two fully connected layers;

[0048] Then, the compact features are sent into the full connection layer after being adaptively adjusted by the time-sensitive factor alpha, and F′ is obtained LI = ((1-alpha) o FC{F LI}) o (alpha o FC{F L}); where FC represents the full connection layer; o is an element-wise multiplication operation; o is an element-wise addition operation; alpha is a time-sensitive factor;

[0049] F L ′ I is a point-to-point registration feature adjustment weight; tanh is a tanh activation function, and sigmoid is a sigmoid activation function; W map is a point-to-point registration feature weight map; F″ LI is an adjusted point-to-point registration feature; is a feature splicing operation; F″′ LI is a global fusion feature;

[0050] Then, F′ LI is compressed into a weight map W map , and F″ LI is obtained by multiplying W map and F′ LI ; where F″ LI = W map o F′ LI , W map = sigmoid(FC(tanh(F′ LI ))); tanh is a tanh activation function, and sigmoid is a sigmoid activation function;

[0051] Finally, after scale judgment by the scene scale factor beta, the feature splicing is performed to obtain the global fusion feature;

[0052] Step 3.2, the global fusion feature F″′ LI is input into the three-dimensional region generation network to generate the region of interest, and after the region feature of the region of interest is extracted, it is sent into the three-dimensional target detection network to complete the detection;

[0053] Further, in the step 3.1, the calculation method of the time-sensitive factor alpha is as follows:

[0054] (1) The input image data is converted from the RGB color space to the YCbCr color space;

[0055] (2) The calculation formula of the time-sensitive factor alpha is as follows:

[0056]

[0057] Wherein Y+Cb+Cr represents the total number of Y, Cb and Cr three components in the input image data and the prior image, for normalization; Y cur_χ , Cb cur_χ , Cr cur_χ Represents the number of Y, Cb and Cr three components in the input image data; Y bas_χ , Cb bas_χ , Cr bas_χ Represents the number of Y, Cb and Cr three components in the prior image.

[0058] Further, in the step 3.1, the calculation process of the scene scale factor β is as follows:

[0059] Firstly, the input image data is segmented using the U-Net++ network to obtain the category to which each pixel point in the input image data belongs;

[0060] Then, the point cloud data is projected onto the segmentation result map output by the U-Net++ network, and the K nearest neighbor method is used to determine the category to which the projection point belongs;

[0061] Finally, the scene scale factor is calculated according to the number of projection points of each category, and the calculation formula is as follows:

[0062]

[0063] Wherein β max is the number of projection points owned by the category with the most projection points, and β min is the number of projection points owned by the category with the least projection points.

[0064] Compared with the prior art, the technical scheme of the present application has the following technical effects:

[0065] The present application realizes more detailed fusion of point cloud and image data, obtains more detailed information of the target, and improves the accuracy of three-dimensional target detection; at the same time, the time factor and the scene scale factor are proposed for different scenes, so that the algorithm has better robustness in complex scenes, and has good application prospect in intelligent driving scene. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 Is the flow chart of the scene adaptive three-dimensional target detection method of the present application radar vision sensor fusion;

[0067] Figure 2 Is a fusion region segmentation schematic diagram;

[0068] Figure 3 Is a global feature fusion network. DETAILED DESCRIPTION

[0069] The application will be further described in detail below in combination with the drawings and specific embodiments.

[0070] As shown in the figure, the scene adaptive three-dimensional target detection method of radar and vision fusion of the application comprises the following specific steps: Figure 1

[0071] Step 1, the original point cloud features and image features are extracted from the input point cloud data and image data respectively.

[0072] For the point cloud data, first, a point is randomly selected as an initial point, then the farthest point sampling method is used to randomly sample in the input point cloud data, when the sampling is completed, a sphere is formed with each sampled point as the center and a specified radius, all the point clouds contained in the sphere are called a cluster, finally the features of each point cloud in the cluster are extracted and maximum pooling is performed in the channel dimension to obtain the cluster features, and all the cluster features are combined to obtain the original point cloud features.

[0073] For the image data, four convolutional blocks in cascade are used for feature extraction, each convolutional block is composed of two 3x3 convolutional layers, followed by a batch normalization layer and a ReLU activation function; then four parallel transposed convolutions with different strides are used to restore the feature maps of different scales to the same resolution, and then they are spliced to obtain image features containing more rich image semantic information.

[0074] Step 2, the input point cloud data is projected onto the image features through matrix transformation, and the point-by-point feature fusion of the point cloud projection points is performed to obtain the point-by-point fusion features of the point cloud.

[0075] Step 2.1, the point cloud filtering processing is performed on the original point cloud data.

[0076] Step 2.2, the filtered point cloud is projected onto the image features according to the following formula:

[0077] P i ′=K r [R c |T c ]P i

[0078] Where P i ′ is the point cloud projection point of the ith point cloud data P i on the image features; K r is the camera intrinsic matrix; R c is the rotation matrix, and T c is the projection matrix.

[0079] Step 2.3, according to the discrete coefficient based on the point cloud projection point, the point-by-point fusion region of the projection point is calculated, which is as follows: ​

[0080] Definition of the ith point cloud data P i Point cloud projection point P on the image feature i Discrete coefficient of P The calculation formula is as follows:

[0081]

[0082] Where d ij is the distance between p i ' and the jth point cloud data P j Point cloud projection point p j ' on the image feature; L is the number of point cloud projection points on the image; H and W represent the height and width of the input image data, respectively; (x i ,y i ), (x j ,y j ) are the coordinates of p i ' and p j ', respectively; the discrete coefficient set consisting of the discrete coefficients of the point cloud projection points is obtained by repeating the above calculation

[0083] Further, the discrete coefficient of each point cloud projection point is normalized and encoded to obtain the weight coefficient of each point cloud projection point, and the point-by-point fusion area of the point cloud projection point is calculated using the weight coefficient, and the calculation formula is as follows:

[0084]

[0085] Where φ i is the weight coefficient corresponding to the projection point p i '; and represent the minimum and maximum values in the discrete coefficient set, respectively; h i and w i are the height and width of the point-by-point fusion area when p i ' is the center point; is the ceiling function.

[0086] Step 2.4, after obtaining the point-by-point fusion area of the projection point, the point-by-point fusion area is divided into I rows and J columns, at this time I×J equal-sized grids are obtained, the center point of each grid is calculated, and the calculation formula is as follows:

[0087]

[0088] Where (u ab ,v ab) is the coordinate of the center point of the grid in the ath row and the bth column in the point-by-point fusion region; a = 1, 2, …, I, b = 1, 2, …, J; h is the height of the point-by-point fusion region, and w is the width of the point-by-point fusion region; (u, v) is the coordinate of the point cloud projection point in the point-by-point fusion region.

[0089] Step 2.5, respectively, calculating the result f ab of the bilinear interpolation and the result f ab of the nearest neighbor interpolation of the center point (u ab_1 , v ab_2 ) in each grid, the calculation process is as follows:

[0090] f ab_1 = f1(u-u1)(v-v1) + f2(u-u1)(v2-v) + f3(u2-u)(v-v1) + f4(u2-u)(v2-v)

[0091]

[0092] (u1, v1), (u1, v2), (u2, v1), (u2, v2) are the coordinates of the four nearest neighbor point cloud projection points of the center point of the grid, respectively; {d1, d2, d3, d4} are the distances from the four nearest neighbor point cloud projection points to the center point of the grid; f1, f2, f3, f4 are the feature vectors corresponding to the four nearest neighbor point cloud projection points, is the feature vector corresponding to the nearest neighbor point cloud projection point of the center point of the grid.

[0093] Step 2.6, the bilinear interpolation result and the nearest neighbor interpolation result of the center point of each grid are fused by weighting to obtain the interpolation feature vector of the corresponding grid, which is as follows:

[0094] The aggregation coefficient is defined as follows:

[0095]

[0096] where N d is the length of the feature vector dimension; f ω_ψ , ω ∈ {1, 2, 3, 4}, ψ ∈ {1, 2, …, N d} represents the feature value of the ωth feature vector f ω in the ψth dimension; μ ψ is the mean value of the value in the ψth dimension; σ ψ is the standard deviation of the value in the ψth dimension; is the aggregation coefficient corresponding to the ψth dimension.

[0097] The interpolation feature vector f for each cell is obtained by weighting and fusing the results of bilinear interpolation and nearest neighbor interpolation based on the aggregation coefficient. ab The calculation formula is as follows:

[0098]

[0099] Step 2.7, based on the position of the center point of the grid (u) ab ,v ab Calculate the center distance metric η for each cell. ab And normalization is achieved, as follows:

[0100] The center distance metric η corresponding to the cell in row a and column b within the point-by-point fusion region ab and its normalization result η ab The formula for calculating ′ is as follows:

[0101]

[0102] where min(u,u ab ) represents u ab The smaller of u and max(u,u) ab ) represents u ab The larger of u and min(v,v) ab ) represents v and v ab The smaller value in the middle, max(v,v) ab ) represents v and v ab The larger value; η ab ′ is the normalized center distance metric coefficient; (u,v) are the coordinates of the point cloud projection points within the point-by-point fusion region.

[0103] Step 2.8: Obtain the interpolated feature vector f for each cell based on the normalized center distance metric coefficients. ab After performing a weighted operation, the point fusion features corresponding to the projected points are obtained, as shown in the following formula:

[0104]

[0105] Where f LI The point fusion feature is the point cloud projection points.

[0106] Step 2.9: Combine the point fusion features corresponding to all point cloud projection points to obtain the point-by-point fusion feature F. LI .

[0107] Step 3, constructing a scene-driven three-dimensional target detection network, proposing a time factor and a scene scale factor according to the weather and the size of the target in the current scene, and realizing adaptive global feature fusion of point-by-point fusion features and original point cloud features according to the time factor and the scene scale factor, and inputting the global fusion features into a three-dimensional region generation network to generate a region of interest, extracting the region features and sending them into a three-dimensional target detection network to complete three-dimensional target detection.

[0108] Compared with the traditional RGB color space, the YCbCr color space separates the brightness and color information, and can better judge the quality of the current scene image, so the image data is converted from the RGB color space to the YCbCr color space, and the calculation formula is as follows:

[0109]

[0110] Wherein R, G, B represent the red, green and blue components in the RGB color space; Y, Cb, Cr represent the brightness component, blue component and red component in the YCbCr color space.

[0111] Generally speaking, the image quality output by the camera is relatively high in the daytime and suitable light weather, so this paper selects an image in the daytime and suitable light weather as a prior image as prior knowledge, and the greater the change of the image output by the current scene (i.e. the aforementioned input image data) relative to the prior image, the lower the image quality of the current scene, which needs to adjust the point-by-point fusion features adaptively through the original point cloud features, so a time factor a is defined to judge the image quality, and the calculation formula is as follows:

[0112]

[0113] Wherein Y+Cb+Cr represents the total number of Y, Cb and Cr components in the current scene image and the prior image, which is used for normalization; Y cur_i , Cb cur_i , Cr cur_i represents the number of Y, Cb, and Cr components with a value of i in the current image; Y bas_i , Cb bas_i , Cr bas_i represents the number of Y, Cb, and Cr components with a value of i in the prior image; when a is larger, it means that the current scene image changes greatly relative to the prior image, and the current scene image quality is worse; when a is smaller, it means that the current scene image quality is better.

[0114] However, even in a good weather scenario, when the target in the scene is small or far away, the target point cloud data collected by even high-precision laser radar will be too sparse, and in this case the information of small-scale targets is less or has disappeared in the original point cloud features. In contrast, the point-by-point fusion features can provide more effective semantic information of small-scale targets. At this time, the original point cloud features may introduce interference information in the global feature fusion process. Therefore, this section defines a scene scale factor β to judge the scale of the target in the current scene, and the specific process is as follows:

[0115] First, the U-Net++ network is used to perform semantic segmentation on the current scene image to obtain the category to which each pixel point in the image belongs. Then, the point cloud data is projected onto the segmentation result image output by the U-Net++ network, and the K-nearest neighbor method is used to determine the category to which the projected point belongs. Finally, the scene scale factor is calculated according to the number of projected points of each category, and the calculation formula is as follows:

[0116]

[0117] Where β max is the number of projected points owned by the category with the most projected points, and β min is the number of projected points owned by the category with the fewest projected points. The smaller β is, the greater the scale difference between different targets, and the less effective information in the original point cloud features. At this time, the proportion of the original point cloud features in the global feature fusion should be smaller. The larger β is, the smaller the scale difference between different targets, and the more effective information in the original point cloud features. The proportion of the original point cloud features in the global feature fusion should be larger.

[0118] By defining the above two factors, the effectiveness of point cloud and image data in the current scene can be determined, and the point cloud-image global feature fusion can be realized according to the diurnal inductive factor and the scene scale factor, and the calculation formula is as follows:

[0119] F′ LI =((1-α)⊙FC{F LI})⊕(α⊙FC{F L})

[0120] W map =sigmoid(FC(tanh(F′ LI )))

[0121] F″ LI =W map ⊙F′ LI

[0122]

[0123] wherein FC is a full connection layer; is an element-wise multiplication operation; is an element-wise addition operation; a is a time-sensitive factor; F′ LI is a point-wise registration feature adjustment weight; tanh is a tanh activation function, and sigmoid is a sigmoid activation function; W map is a point-wise registration feature weight map; F″ LI is an adjusted point-wise registration feature; is a feature stitching operation; F″′ LI is a global fusion feature.

[0124] In the global feature fusion network, firstly, the two features are mapped to the same channel dimension through two full connection layers, then the point-wise registration features are adaptively adjusted through the time-sensitive factor a, after that, they are added together to form a compact feature, which is sent to a full connection layer, and is compressed into a weight map, then the adjusted point-wise registration feature is obtained by multiplying the point-wise registration feature with the weight map, finally, the feature is multiplied with the scene scale factor β to judge the importance of the feature, then the feature stitching is performed to complete the global feature fusion.

[0125] Finally, the global fusion feature is input into the three-dimensional region generation network to generate the region of interest, the region feature is extracted and sent into the three-dimensional target detection network to complete the three-dimensional target detection.

[0126] The above examples only illustrate the technical idea of the present application, and cannot limit the protection scope of the present application, any modification made according to the technical idea of the present application on the basis of the technical scheme falls within the protection scope of the present application.

Claims

1. A scene adaptive 3D target detection method for radar and vision sensor fusion, characterized in that, The specific process is as follows: Step 1, the input point cloud data and image data are extracted respectively to obtain the original point cloud feature and image feature; Step 2, the input point cloud data is projected onto the image feature, and the point cloud projection point is fused point by point to obtain the point cloud data point by point fusion feature; Step 3, a scene driven three-dimensional target detection network is constructed, and the trained three-dimensional target detection network is used for real-time detection of the target; The specific process of step 3 is as follows: Step 3.1, the point-wise fusion feature F LI and the original point cloud feature F L input the global feature fusion network for global feature fusion: First, F LI and F L are mapped into the same channel dimension by two fully connected layers; Then, the compact features are fed into the fully connected layer by adaptively adjusting the time-sensitive factor a and adding them together to obtain wherein FC represents the fully connected layer; and is an element-wise multiplication operation; is an element-wise addition operation; and a is a time-sensitive factor. F′ LI is the point-wise registration feature adjusted by the weight; tanh is the tanh activation function, and sigmoid is the sigmoid activation function; W map is the point-wise registration feature weight map; F″ LI is the adjusted point-wise registration feature; is the feature stitching operation; F″′ LI is the global fused feature; Then, F′ LI Compressed into a weighted graph W map W map With F′ LI Multiplying them together gives F″ LI ; where F″ LI =W map ⊙F′ LI W map =sigmoid(FC(tanh(F′) LI tanh is the tanh activation function, and sigmoid is the sigmoid activation function. Finally, after the scale judgment by the scene scale factor β, the feature splicing is carried out to obtain the global fusion feature; Step 3.2, the global fusion feature F" LI After the three-dimensional region generation network generates the region of interest and extracts the region feature of the region of interest, the three-dimensional target detection network is used for detection. In step 3.1, the calculation method of the weather sensing factor α is as follows: (1) the input image data is converted from RGB color space to YCbCr color space; (2) the calculation formula of the weather sensing factor α is as follows: where Y+Cb+Cr represents the total number of Y, Cb and Cr components in the input image data and the prior image, for normalization; Y cur_χ , Cb cur_χ , Cr cur_χ represents the number of Y, Cb and Cr components in the input image data that take the value χ; Y bas_χ , Cb bas_χ , Cr bas_χ represents the number of Y, Cb and Cr components in the prior image that take the value χ; In step 3.1, the calculation process of the scene scale factor β is as follows: Firstly, the input image data is segmented by using the U-Net++ network to obtain the category of each pixel point in the input image data; Then, the point cloud data is projected onto the segmentation result map output by the U-Net++ network, and the K nearest neighbor method is used to determine the category of the projection point; Finally, the scene scale factor is calculated according to the number of projection points of each category, and the calculation formula is as follows: where β max is the number of projection points owned by the class with the most projection points, β min is the number of projection points owned by the class with the fewest projection points.

2. The method of claim 1, wherein the method further comprises: The specific process of extracting the original point cloud feature from the input point cloud data in step 1 is as follows: For point cloud data, first randomly select a point cloud data as the initial point cloud data; then randomly sample in the point cloud data by using the farthest point sampling method, when the sampling is completed, take each sampled point cloud data as the center of a sphere, and specify the radius to form a sphere, all the point cloud data contained in each sphere is called a cluster; finally, the features of each point cloud data in each cluster are extracted, and the channel dimension is maximum pooled to obtain the cluster feature, and all the cluster features are combined to obtain the original point cloud feature.

3. The method of claim 1, wherein the method further comprises: The specific process of extracting the image feature from the input image data in step 1 is as follows: For image data, first input four convolution blocks in cascade, and each convolution block is followed by a batch normalization layer and a ReLU activation function; then four different step size transposed convolutions are used in parallel on the output of the last convolution block to obtain four feature maps with the same resolution, and the four feature maps with the same resolution are spliced to obtain the image feature.

4. The method of claim 1, wherein the method further comprises: The specific process of step 2 is as follows: Step 2.1, project the input point cloud data onto the image feature; Step 2.2, divide the point cloud projection point into a point by point fusion area: where h i and w i are the height and width of the point-wise fusion region respectively, which is divided with the point cloud projection point P i ′ on the image feature as the center point; i is the floor function; φ i is the weight coefficient of P i ′, and is the discrete coefficient of P i ′, and respectively represent the minimum and maximum values in the discrete coefficient set ; H and W respectively represent the height and width of the image data; i, j ∈ [1, 2, …, L], L is the number of point cloud projection points; d ij is the distance between p′ i and the jth point cloud data P j point cloud projection point p′ j on the image feature.​ Step 2.3, divide the point cloud projection point into a point by point fusion area: Step 2.4, for each grid center point, obtain the bilinear interpolation result and the nearest neighbor interpolation result; Step 2.5, weight fusion is carried out on the bilinear interpolation result and the nearest neighbor interpolation result of each grid center point to obtain the interpolation feature vector of the corresponding grid; Step 2.6, calculate the center distance measurement coefficient of each grid, and normalize it; Step 2.7, the interpolation feature vectors of each grid are weighted and summed according to the normalized center distance measurement coefficient to obtain the point fusion features of the corresponding point cloud projection points; Step 2.8, the point fusion features of all point cloud projection points are spliced to obtain the point-by-point fusion features of the point cloud data.

5. The method of claim 4, wherein the method further comprises: In step 2.4, the result f of the bilinear interpolation of the center point of the grid in the ath row and bth column in the fusion region is fused point by point ab_1 and the result f of the nearest neighbor interpolation ab_2 The calculation process is as follows: f ab_1 = f1(u - u1)(v - v1) + f2(u - u1)(v2 - v) + f3(u2 - u)(v - v1) + f4(u2 - u)(v2 - v) wherein (u1, v1), (u1, v2), (u2, v1), (u2, v2) are coordinates of the four nearest neighbor point cloud projection points of the grid center point, respectively; d1, d2, d3, d4 are distances from the four nearest neighbor point cloud projection points to the grid center point; f1, f2, f3, f4 are feature vectors corresponding to the four nearest neighbor point cloud projection points, is the feature vector corresponding to the nearest neighbor point cloud projection point of the grid center point.

6. The method of claim 4, wherein the method further comprises: In step 2.5, the interpolation eigenvector f of the grid cell in the a-th row and the b-th column in the fusion region is fused point by point ab The calculation formula is as follows: where f ab_1 and f ab_2 are the results of bilinear interpolation and nearest neighbor interpolation of the center point of the a-th row and b-th column grid in the fusion region, respectively; is the aggregation coefficient, N d is the length of the feature vector dimension, f ω_ψ represents the feature value of the ω-th feature vector on the ψ-th dimension; μ ψ is the mean value of the values on the ψ-th dimension; σ ψ is the standard deviation of the values on the ψ-th dimension.

7. The method of claim 4, wherein the method further comprises: In step 2.7, the point fusion feature f of the projection point of the point cloud corresponding to the grid pair of the a-th row and the b-th column in the fusion region is fused point by point LI The calculation formula is as follows: wherein f ab is the interpolation feature vector of the grid at the ath row and bth column in the point-by-point fusion region, η ab is the normalized result of the center distance measurement coefficient corresponding to the grid at the ath row and bth column in the point-by-point fusion region, (u ab ,v ab ) is the coordinate of the center point of the grid at the ath row and bth column in the point-by-point fusion region; min(u,u ab ) represents the smaller value of u ab and u, max(u,u ab ) represents the larger value of u ab and u; min(v,v ab ) represents the smaller value of v and v ab , max(v,v ab ) represents the larger value of v and v ab , and (u,v) is the coordinate of the point cloud projection point in the point-by-point fusion region.

Images