Image and point cloud fusion method and device

Abstract

The application relates to an image and point cloud fusion method, comprising the following steps: obtaining an image detection result and a point cloud detection result, wherein the point cloud detection result is obtained by fusing image and point cloud data; matching and associating the image detection result and the point cloud detection result belonging to the same object; inputting a plurality of point cloud detection results into a neural network model to obtain a plurality of confidence scores corresponding to the point cloud detection results; and obtaining a fusion detection result according to the image detection result, the point cloud detection result and the confidence scores, so as to solve the problem that the prior art cannot effectively and accurately detect a target object.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to an image and point cloud fusion method and device. BACKGROUND

[0002] The key of three-dimensional object detection in the field of automatic driving is to identify and locate decision-sensitive objects in a three-dimensional environment. An automatic driving vehicle is usually equipped with a millimeter wave radar and multiple surround-view cameras to provide omnidirectional perception capability. However, in the related art, three-dimensional object detection is performed by using data captured by a single sensor. Due to the reasons of point cloud sparsity, large azimuth angle error and low measurement surface accuracy of the millimeter wave radar, the recall rate of three-dimensional target detection is low, and the target object cannot be accurately detected. For three-dimensional object detection based on a camera, since the RGB image cannot directly provide distance and speed information, some studies map two-dimensional features to a three-dimensional space through a depth estimation method, and then perform object detection in the three-dimensional space, and simultaneously use a simple regression network to predict the speed. However, the speed prediction error is also large. Therefore, the above two three-dimensional object detection methods cannot effectively and accurately detect the target object. SUMMARY

[0003] One of the purposes of the present application is to provide an image and point cloud fusion method to solve the problem that the prior art cannot effectively and accurately detect the target object. The second purpose is to provide an image and point cloud fusion device.

[0004] In order to achieve the above purpose, the technical scheme adopted by the present application is as follows:

[0005] An image and point cloud fusion method comprises: obtaining an image detection result and a point cloud detection result, wherein the point cloud detection result is obtained by fusing image and point cloud data; matching and associating the image detection result and the point cloud detection result belonging to the same object; inputting a plurality of point cloud detection results into a neural network model to obtain a plurality of confidence scores corresponding to the point cloud detection results; and obtaining a fusion detection result according to the image detection result, the point cloud detection result and the confidence scores.

[0006] According to the above technical means, the point cloud data collected by the millimeter wave radar and the image collected by the camera are used to obtain image detection results and point cloud detection results; the image detection results and the point cloud detection results belonging to the same object are matched and associated; a plurality of point cloud detection results are input into a neural network model to obtain a plurality of confidence scores corresponding to the point cloud detection results; and a fusion detection result is obtained according to the image detection result, the point cloud detection result, and the confidence score. In this way, the millimeter wave radar can provide high angular resolution and ranging accuracy, provide accurate speed information and a wide field of view, thereby enabling all-around environmental perception, and the image can provide rich color, texture, and semantic information at high resolution. The advantages of the two are combined to achieve more accurate and comprehensive three-dimensional object detection capability, and the perception accuracy and reliability of the automatic driving system are improved.

[0007] Further, before the image detection result and the point cloud detection result are obtained, the method further comprises: screening the to-be-acquired image detection result and the to-be-acquired point cloud detection result to obtain the image detection result and the point cloud detection result.

[0008] According to the above technical means, the image detection result and the point cloud detection result with higher detection scores are screened, and the accuracy of detection perception is improved.

[0009] Further, the screening of the to-be-acquired image detection result and the to-be-acquired point cloud detection result to obtain the image detection result and the point cloud detection result comprises: comparing an image detection score of the to-be-acquired image detection result with a set image detection threshold value, and if the image detection score is greater than the image detection threshold value, the image detection result is obtained; and comparing a point cloud detection score of the to-be-acquired point cloud detection result with a set point cloud detection threshold value, and if the point cloud detection score is greater than the point cloud detection threshold value, the point cloud detection result is obtained.

[0010] The above technical means screen the image detection result and the point cloud detection result with higher detection scores, and improve the accuracy of detection perception.

[0011] Further, the matching and associating the image detection result and the point cloud detection result belonging to the same object comprises: judging whether the first label of the image detection result and the second label of the point cloud detection result are the same; if the first label and the second label are the same, obtaining the first projection center of the image detection result and the depth of the image detection result; calculating the second projection center of the point cloud detection result and the depth of the point cloud detection result; calculating the distance between the first projection center and the second projection center and the difference value of the depth of the image detection result and the depth of the point cloud detection result; and determining whether to match and associate the image detection result and the point cloud detection result according to the distance and the difference value.

[0012] The above technical means screens out the image detection result and the point cloud detection result belonging to the same object, and realizes the detection of the same object.

[0013] Further, the calculating the second projection center of the point cloud detection result and the depth of the point cloud detection result comprises: determining the second projection center according to the first projection center and the bias of the point cloud detection result relative to the second projection center; and determining the depth of the point cloud detection result according to the depth information of the image detection result and the depth bias of the point cloud detection result.

[0014] The above technical means screens out the image detection result and the point cloud detection result belonging to the same object, and realizes the detection of the same object.

[0015] Further, the obtaining the fusion detection result according to the image detection result, the point cloud detection result and the confidence score comprises: judging the confidence score; if the confidence score is greater than a set confidence threshold, determining the depth of the fusion detection result according to the depth of the point cloud detection result and the confidence score; determining the projection center of the fusion detection result according to the projection center of the point cloud detection result and the confidence score; and if the confidence score is less than or equal to the set confidence threshold, taking the image detection result as the fusion detection result.

[0016] According to the above technical means, more accurate and comprehensive three-dimensional object detection capability is realized, and the perception accuracy and reliability of the automatic driving system are improved.

[0017] The image and point cloud fusion device comprises: an acquisition module configured to acquire image detection results and point cloud detection results, wherein the point cloud detection results are obtained by fusing image and point cloud data; an association module configured to match and associate the image detection results and the point cloud detection results belonging to the same object; a confidence module configured to input a plurality of the point cloud detection results into a neural network model to obtain a plurality of confidence scores corresponding to the point cloud detection results; and a fusion module configured to obtain fusion detection results according to the image detection results, the point cloud detection results and the confidence scores.

[0018] Further, the device further comprises a screening module configured to screen the image detection results and the point cloud detection results to be acquired to obtain the image detection results and the point cloud detection results.

[0019] Further, the screening module comprises a comparison unit configured to compare an image detection score of the image detection results to be acquired with a set image detection threshold value, and if the image detection score is greater than the image detection threshold value, the image detection results are obtained.

[0020] Further, the screening module comprises a comparison unit configured to compare an image detection score of the image detection results to be acquired with a set image detection threshold value, and if the image detection score is greater than the image detection threshold value, the image detection results are obtained.

[0021] Further, the association module comprises:

[0022] a first judgment unit configured to judge whether a first label of the image detection results and a second label of the point cloud detection results are the same;

[0023] an obtaining unit configured to obtain a first projection center of the image detection results and a depth of the image detection results if the first judgment unit judges that the first label and the second label are the same;

[0024] a first calculation unit configured to calculate a second projection center of the point cloud detection results and a depth of the point cloud detection results;

[0025] a second calculation unit configured to calculate a distance between the first projection center and the second projection center and a difference value between the depth of the image detection results and the depth of the point cloud detection results;

[0026] an association unit configured to determine whether to match and associate the image detection results and the point cloud detection results according to the distance and the difference value.

[0027] Further, the first calculation unit comprises a projection center submodule, configured to determine the second projection center according to the first projection center and the bias of the point cloud detection result relative to the second projection center.

[0028] a depth submodule, configured to determine the depth of the point cloud detection result according to the depth information of the image detection result and the depth bias of the point cloud detection result.

[0029] Further, the fusion module comprises a second judgment unit, configured to judge the confidence score; a depth unit, configured to, if the judgment result of the second judgment unit is that the confidence score is greater than a set confidence threshold, determine the depth of the fusion detection result according to the depth of the point cloud detection result and the confidence score; a projection center unit, configured to determine the projection center of the fusion detection result according to the projection center of the point cloud detection result and the confidence score; and a fusion unit, configured to, if the judgment result of the second judgment unit is less than or equal to the set confidence threshold, take the image detection result as the fusion detection result.

[0030] The present application has the following advantages:

[0031] The present application obtains an image detection result and a point cloud detection result, wherein the point cloud detection result is obtained by fusing image and point cloud data; the image detection result and the point cloud detection result belonging to the same object are matched and associated; a plurality of confidence scores corresponding to the point cloud detection result are obtained by inputting a plurality of point cloud detection results into a neural network model; and a fusion detection result is obtained according to the image detection result, the point cloud detection result and the confidence score, that is, the present application enhances the millimeter wave radar feature by using image features, realizes image-radar association at the detection level by using the enhanced millimeter wave radar feature, and improves the depth estimation and projection center of multi-camera vision by fusing the enhanced millimeter wave radar depth, so as to obtain a fusion detection result which comprehensively combines the advantages of image and point cloud data, thereby realizing more accurate and comprehensive three-dimensional object detection capability and improving the perception accuracy and reliability of the automatic driving system. BRIEF DESCRIPTION OF DRAWINGS

[0032] Figure 1 Fig. 1 is a network architecture diagram of image and point cloud fusion according to the present application;

[0033] Figure 2 Fig. 2 is a flowchart of an image and point cloud fusion method according to the present application;

[0034] Figure 3 Fig. 3 is a flowchart of an image and point cloud fusion method according to the present application;

[0035] Figure 4A structural schematic diagram of an image and point cloud fusion device according to the present application. DETAILED DESCRIPTION

[0036] Other advantages and effects of the present application can be easily understood by those skilled in the art from the disclosure of the present specification. The present application can also be implemented or applied by means of other different specific embodiments, and the details in the present specification can be modified or changed based on different views and applications without departing from the spirit of the present application. It should be understood that the preferred embodiments are only for illustrating the present application, but not for limiting the protection scope of the present application.

[0037] Reference is made to Figure 1 which shows a network architecture schematic diagram for implementing an image and point cloud fusion method according to the present application, and specifically includes:

[0038] A camera, a millimeter wave radar, a 2D convolution backbone network, a 2D sparse convolution backbone network, a 3D detection network and a neural network model arranged on a vehicle.

[0039] 2D visual images are collected by the cameras arranged on the vehicle, the 2D visual images obtained by the cameras arranged on the vehicle are respectively encoded by the 2D convolution backbone network, 2D image coding features are obtained, then, the 2D image coding features are converted into front-view 3D grid features with the camera as the origin by depth estimation and projection according to the depth discrete confidence, and finally, the 3D grid features are converted to the center of the coordinate system with the origin of the vehicle coordinate system as the center to generate a unified BEV perspective 3D grid feature space. Figure 3 The 3D grid feature space corresponding to the image is input into the 3D detection network to obtain the image detection result.

[0040] The point cloud data is collected by the millimeter wave radar arranged on the vehicle, the point cloud data is mapped to a 3D grid BEV space, a BEV perspective 3D grid feature space is obtained, a 2D sparse convolution backbone network is used to encode the radar point cloud data, and a radar point cloud feature code is obtained. In the process of encoding the point cloud data, the BEV perspective 3D grid feature obtained from the image is introduced into the 2D sparse convolution backbone network, image plane space information is provided for the millimeter wave radar, that is, the BEV perspective 3D grid feature obtained from the image and the point cloud feature code are taken as inputs of the 2D sparse convolution backbone network, a unified BEV perspective 3D grid feature space corresponding to the point cloud data is generated, the 3D grid feature space corresponding to the point cloud is input into a 3D detection network, and a point cloud detection result is obtained. By using image features, rich color, texture and semantic information can be provided, and by using millimeter wave radar features, high angular resolution and ranging accuracy can be provided, so that the point cloud detection result combines the advantages of image and point cloud data. The image detection result and the point cloud detection result are filtered, the filtered multiple point cloud detection results are input into a neural network model for deep fusion, multiple confidence scores corresponding to the point cloud detection result are obtained, and a fusion detection result is obtained according to the confidence scores, the image detection result and the point cloud detection result.

[0041] Based on the above architecture, the fusion process of the image and the point cloud of the present application will be described in detail below, referring to Figure 2 which shows a flowchart of an image and point cloud fusion method according to an embodiment of the present application, the method comprising:

[0042] Step 201: obtaining an image detection result and a point cloud detection result.

[0043] In actual application, the image detection result is obtained by the following method. Specifically, a 2D convolution backbone network Encode img is used to perform image feature coding on 2D visual images M i (i = 1, 2, …, N) obtained by each perspective camera, respectively, to obtain image coding features of each perspective camera Then, the confidence estimation of each image pixel in a certain depth discrete interval is obtained by depth estimation The features of each pixel are weighted and averaged into the discrete grid points corresponding to each depth, and finally the three-dimensional voxel features corresponding to each perspective camera image are obtained The intrinsic parameters I i (i = 1, 2, …, N) of each perspective camera and the extrinsic parameters The three-dimensional voxel features corresponding to each perspective camera image are transformed to a unified BEV perspective 3D grid feature space with the origin of the vehicle coordinate system as the center through coordinate system conversion The obtained BEV perspective 3D grid feature space is input into a 3D detection network, and a corresponding 3D target detection task result is output , that is, an image detection result, which includes: a projection center of the image , a depth of the image , a classification index of the image , a detection score of the image , and a size and direction of an image 3D detection box, etc.

[0044] Among them, the intrinsic generally refers to the camera X or Y direction center point and focal length, and the extrinsic generally refers to the rotation matrix or translation matrix.

[0045] Specifically, in the step 201, the 2D convolution backbone network Encode 2D assigns the same weight to each perspective camera 2D visual image M i (i = 1, 2, …, N), that is, the weights of the perspective camera 2D visual images are the same. In actual application, the 2D convolution backbone network can adopt any one of the following: ResNet, EfficientNet, SwinTransformer, VoVNetV2, and ConvNeXt, and then be combined with the CBNet backbone network architecture and the feature pyramid (FPN) multi-scale fusion.

[0046] Specifically, the method for image coding feature depth estimation mainly estimates each perspective camera 2D visual image feature map through a depth estimation network, and the prediction result can be the confidence of each image pixel point in the [0, 1] interval . The corresponding true value depth is 1. Other non-true value depth points are 0 supervision; or Gaussian distribution parameters estimated by specifying the mean and standard deviation . The corresponding true value depth is supervised on the mean . Specifically, the 3D detection network includes but is not limited to using: an anchor-based PointPillars detection head, a CenterPoint detection head based on center heat map prediction, a DETR3D based on Transformer, and a Transfusion-based detection head.

[0047] In actual application, the point cloud detection result is obtained in the following manner. Specifically, in combination with the extrinsic parameters of each millimeter wave radar relative to the vehicle body coordinate system , the millimeter wave radar point cloud data L j(j = 1, 2, …, M) coordinate conversion to a unified BEV perspective 3D grid feature space centered on the vehicle coordinate system using a 2D sparse convolution backbone network Encode radar , respectively, the radar point cloud data obtained by the millimeter wave radar is encoded. At the same time of encoding, the obtained image 3D grid features are also input into Encode radar , at each layer of the 2D sparse convolution backbone network, the aligned image 3D grid features and millimeter wave radar 3D grid features are concatenated to obtain 3D resolution level fusion features. Then, through an independent neck network composed of a feature pyramid network (FPN), the multi-layer features are fused. Finally, the obtained fusion features are input into a 3D detection network, and the corresponding 3D target detection task results obtained by the millimeter wave radar are output , i.e. point cloud detection results. The point cloud detection results include: point cloud projection center bias point cloud depth bias point cloud classification index and point cloud detection score , etc.

[0048] Specifically, in step 201, the 2D sparse convolution backbone network Encode radar is similar to Encode img , in actual application, the 2D sparse convolution backbone network can adopt any one of the following: ResNet, EfficientNet, SwinTransformer, VoVNetV2, ConvNeXt, and then combined with CBNet backbone network architecture and feature pyramid (FPN) multi-scale fusion.

[0049] Specifically, the 3D detection network includes but is not limited to using: anchor-based PointPillars detection head, CenterPoint detection head based on center heat map prediction, DETR3D based on Transformer, and Transfusion-based detection head.

[0050] Step 202: matching and associating the image detection results and the point cloud detection results belonging to the same object.

[0051] In actual application, whether the image detection result and the point cloud detection result belong to the detection of the same object can be determined according to the classification index, i.e. whether the image target attribute and the point cloud target attribute are the same according to the classification index, if the classification indexes of the image target attribute and the point cloud target attribute are the same, it belongs to the same object, i.e. the detection results obtained by the two modalities have the same class prediction label The image detection result and the point cloud detection result are matched and associated.

[0052] If the classification indexes of the image detection result and the point cloud detection result are not the same, it does not belong to the same object, and matching and association is not performed.

[0053] The same object can be a pedestrian, a vehicle, etc.

[0054] Step 203: inputting the plurality of point cloud detection results into a neural network model to obtain a plurality of confidence scores corresponding to the point cloud detection results.

[0055] In actual application, it is assumed that a given multi-camera image detection result There are N millimeter wave radar point cloud detection results All the N point cloud detection results are inputted into a neural network model to obtain a series of confidence scores alpha j .

[0056] The neural network model includes, but is not limited to, the following modes: a convolutional neural network module, a deep weighted network DWN.

[0057] The DWN network can select a 4-layer multi-layer perceptron structure or a multi-layer network based on Transformer, and the present application does not make specific limitations.

[0058] Step 204: obtaining a fusion detection result according to the image detection result, the point cloud detection result and the confidence score.

[0059] In the embodiment, the image detection result and the point cloud detection result are obtained; the image detection result and the point cloud detection result belonging to the same object are matched and associated; the point cloud detection result is inputted into a neural network model to obtain a confidence score corresponding to the point cloud detection result; and a fusion detection result is obtained according to the image detection result, the point cloud detection result and the confidence score. The present application enhances the millimeter wave radar feature by using the image feature, realizes image-radar association at the detection level by using the enhanced millimeter wave radar feature, and improves the depth estimation and projection center of multi-camera vision by fusing the enhanced millimeter wave radar depth, so as to obtain a fusion detection result. The fusion detection result combines the advantages of image and point cloud data, so as to realize more accurate and comprehensive three-dimensional object detection capability, and improve the perception accuracy and reliability of the automatic driving system.

[0060] Referring to Figure 3 , a flowchart of a method for fusing an image and a point cloud according to an embodiment of the present application is shown, and the method comprises:

[0061] Step 301: screening the image detection result to be obtained and the point cloud detection result to be obtained, to obtain the image detection result and the point cloud detection result.

[0062] In actual application, the image detection result to be obtained can be obtained in the following manner. Specifically, a 2D convolution backbone network Encode img is used to respectively perform image feature coding on 2D visual images M i (i=1, 2, …, N) obtained by each perspective camera, to obtain image coding features of each perspective camera. Then, a confidence estimation of each image pixel point in a certain depth discrete interval is obtained through depth estimation. The features of each pixel are weighted and averaged into the corresponding discrete grid points of each depth, and finally the three-dimensional voxel features corresponding to each perspective camera image are obtained. In combination with the internal parameters I i (i=1, 2, …, N) of each perspective camera and the external parameters of the relative vehicle coordinate system, the three-dimensional voxel features corresponding to each perspective camera image are transformed to a unified BEV perspective 3D grid feature space with the origin of the vehicle coordinate system as the center. The obtained BEV perspective 3D grid feature space is input into a 3D detection network, and a corresponding 3D target detection task result is output. That is, the image detection result to be obtained, which includes the detection score of the image and the like.

[0063] The detection score of the image is compared with a set image detection threshold. If the image detection score is greater than the image detection threshold , the image detection result is obtained, and if the image detection score is less than the image detection threshold, the image detection result is discarded.

[0064] In actual application, the point cloud detection result to be obtained can be obtained in the following manner. Specifically, in combination with the external parameters of the relative vehicle coordinate system of each millimeter wave radar , the coordinates of millimeter wave radar point cloud data L j (j=1, 2, …, M) are converted to a unified BEV perspective 3D grid feature space with the vehicle coordinate system as the center. A 2D sparse convolution backbone network Encode radar is used to respectively perform point cloud feature coding on the radar point cloud data obtained by the millimeter wave radar. At the same time of coding, the obtained image 3D grid features are also input into Encode radarIn the 2D sparse convolution backbone network, the aligned image 3D grid features and the millimeter wave radar 3D grid features are concatenated at each layer of the 2D sparse convolution backbone network, so as to obtain three resolution levels of fusion features. Then, a separate neck network composed of a feature pyramid network (FPN) is used to fuse the multi-layer features. Finally, the obtained BEV fusion features are input into a 3D detection network, and the corresponding 3D target detection task results obtained by the millimeter wave radar are output That is, a to-be-acquired point cloud detection result. The to-be-acquired point cloud detection result includes a detection score of a point cloud and so on.

[0065] The detection score of the point cloud is compared with a set point cloud detection threshold value. If the detection score of the point cloud is greater than the point cloud detection threshold value the point cloud detection result is obtained, and if the detection score of the point cloud is less than the point cloud detection threshold value, the point cloud detection result is discarded.

[0066] Step 302: matching and associating the image detection result and the point cloud detection result belonging to the same object.

[0067] The matching and associating of the image detection result and the point cloud detection result belonging to the same object is to ensure that the camera and the millimeter wave radar detect the same object, that is, to simultaneously detect a person or a vehicle.

[0068] Specifically, step 302 includes the following sub-steps:

[0069] Sub-step 3021: determining whether the first label of the image detection result and the second label of the point cloud detection result are the same. If they are the same, sub-step 3022 is executed, and if they are not the same, sub-step 3026 is executed.

[0070] In actual application, if the first label of the image target attribute and the second label of the point cloud target attribute are the same, it means that they are the same object.

[0071] Sub-step 3022: obtaining a first projection center of the image detection result and a depth of the image detection result.

[0072] The process of obtaining the first projection center of the image detection result and the depth of the image detection result is the same as that of step 201, and the specific process can be parameterized in step 201. Therefore, the present application will not be described here.

[0073] Sub-step 3023: calculating a second projection center of the point cloud detection result and a depth of the point cloud detection result.

[0074] The second projection center is determined according to the first projection center and the bias of the point cloud detection result relative to the second projection center; and the depth of the point cloud detection result is determined according to the depth information of the image detection result and the depth bias of the point cloud detection result.

[0075] The projection center of the millimeter wave radar can be calculated by the following formula and the depth

[0076]

[0077]

[0078] wherein, represents the projection center of the point cloud, represents the depth of the point cloud, represents the projection center of the image, represents the depth of the image represents the projection center bias of the point cloud, represents the depth bias of the point cloud.

[0079] Sub-step 3024: Calculate the distance between the first projection center and the second projection center and the difference value of the depth of the image detection result and the depth of the point cloud detection result.

[0080] Sub-step 3025: Determine whether to match and associate the image detection result and the point cloud detection result according to the distance and the difference value.

[0081] The distance between the first projection center and the second projection center is calculated by the following formula:

[0082]

[0083] Determine whether the distance between the first projection center and the second projection center is less than the distance threshold value, if less than the distance threshold value T p The depth difference value is calculated.

[0084] The difference value of the depth of the image detection result and the depth of the point cloud detection result is calculated by the following formula:

[0085]

[0086] Determine whether the difference value of the depth is less than the depth threshold value, if less than the depth threshold value T d , the image detection result and the point cloud detection result are matched and associated.

[0087] wherein T represents a distance between the first projection center and the second projection center, H represents the depth difference value, T p represents the distance threshold value and T d represents the depth threshold value, wherein the threshold value can be set by any appropriate manner by those skilled in the art, such as setting the threshold value by artificial experience or setting the threshold value according to the difference value of historical data, which is not limited by the present application.

[0088] Sub-step 3026: the target of the image detection result and the target of the point cloud detection result belong to different objects.

[0089] Step 303: inputting the plurality of point cloud detection results into the neural network model to obtain a plurality of confidence scores corresponding to the point cloud detection results.

[0090] wherein as a preferred manner, the neural network model can use a depth weighted network DWN, and the DWN is used for result-level fusion of the matched point cloud detection results, for example: a given multi-camera detection result has N millimeter wave radar detection results and is associated with all these N results to input into the DWN to obtain a series of confidence scores α j .

[0091] The output of the DWN network is the confidence score α j , which is between 0 and 1, wherein the confidence score of 1 indicates that the detection result of the millimeter wave radar is more accurate, and the confidence score of 0 indicates that the detection result of the multi-camera is more accurate.

[0092] When the confidence score is between 0 and 1, a confidence threshold value can be set, when the confidence score is greater than the set confidence threshold value, it is considered that the detection result of the millimeter wave radar is more accurate, and when the confidence score is less than the set confidence threshold value, it is considered that the detection result of the multi-camera is more accurate. The confidence threshold value can be set to 0 or 1, or other values, which are not specifically limited by the present application.

[0093] The input of the DWN network is a 14-element vector composed of DtHead output features, original depth, distance and Doppler / predicted velocity consistency. During training, the true value label is assigned by the depth of the bounding box true value. If the true value depth z of a bounding box is closer to the millimeter wave radar estimated depth, the true value label α j = 1, otherwise 0

[0094]

[0095] wherein, represents the depth of the point cloud, represents the depth of the image, represents the depth of the point cloud fusion.

[0096] Step 304: obtaining a fusion detection result according to the image detection result, the point cloud detection result and the confidence score.

[0097] Step 304 includes the following sub-steps:

[0098] Sub-step 3041: judging the confidence score, if the confidence score is greater than a set confidence threshold, executing sub-step 3042, if the confidence score is less than or equal to the set confidence threshold, executing sub-step 3044.

[0099] Sub-step 3042: determining the depth of the fusion detection result according to the depth of the point cloud detection result and the confidence score.

[0100] Since there are generally multiple millimeter wave radars, when the confidence score is greater than the set confidence threshold, since each point cloud detection result corresponds to a confidence score, the average of the depths of multiple point cloud detection results and the confidence scores is calculated, and the average is taken as the depth of the fusion detection result

[0101] The depth of the fusion detection result is calculated by the following formula

[0102]

[0103] Where T α represents the confidence threshold.

[0104] Sub-step 3043: determining the projection center of the fusion detection result according to the projection center of the point cloud detection result and the confidence score.

[0105] Since there are generally multiple millimeter wave radars, when the confidence score is greater than the set confidence threshold, since each point cloud detection result corresponds to a confidence score, the average of the projection centers of multiple point clouds and the confidence scores is calculated, and the average is taken as the projection center of the fusion detection result

[0106] The projection center of the fusion detection result is calculated by the following formula

[0107]

[0108]

[0109] Where T α represents the confidence threshold.

[0110] Sub-step 3044: taking the image detection result as the fusion detection result.

[0111] When the confidence score is 1, it means that the detection result of the millimeter wave radar is more accurate, at this time, the fusion detection result is obtained through the point cloud detection result and the confidence score, at this time, the fusion detection result utilizes the image features to provide rich color, texture and semantic information and the millimeter wave radar features to provide higher angular resolution and ranging accuracy, thereby realizing more accurate and comprehensive three-dimensional object detection capability and improving the safety of automatic driving.

[0112] When the confidence score is 0, it means that the detection result of the multi-camera is more accurate, at this time, the image collected by the camera is directly taken as the fusion detection result for output.

[0113] In this embodiment, the image detection result and the point cloud detection result are obtained, the image detection result and the point cloud detection result belonging to the same object are matched and associated, the point cloud detection result is input into the neural network model to obtain the confidence score corresponding to the point cloud detection result, and the fusion detection result is obtained according to the image detection result, the point cloud detection result and the confidence score. That is, the image features are used to enhance the millimeter wave radar features in the present application, the enhanced millimeter wave radar features are used to realize image-radar association at the detection level, and the depth of the enhanced millimeter wave radar is fused to improve the depth estimation and the projection center of the multi-camera vision, so as to obtain the fusion detection result. The fusion detection result combines the advantages of image and point cloud data, thereby realizing more accurate and comprehensive three-dimensional object detection capability and improving the perception accuracy and reliability of the automatic driving system. It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all expressed as a series of action combinations, but those skilled in the art should know that the present application is not limited by the described action sequence, because according to the present application, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the involved actions are not necessarily essential to the present application.

[0114] Based on the description of the foregoing method embodiments, the present application also provides corresponding device embodiments to realize the content described in the foregoing method embodiments.

[0115] With reference to Figure 4 which shows a structure schematic diagram of an image and point cloud fusion device according to an embodiment of the present application, the device comprises:

[0116] The acquisition module 401 is configured to obtain an image detection result and a point cloud detection result, wherein the point cloud detection result is obtained by fusing image and point cloud data.

[0117] The association module 402 is configured to match and associate the image detection result and the point cloud detection result belonging to the same object.

[0118] The confidence module 403 is configured to input the plurality of point cloud detection results into a neural network model to obtain a plurality of confidence scores corresponding to the point cloud detection results.

[0119] The fusion module 404 is configured to obtain a fusion detection result according to the image detection result, the point cloud detection result and the confidence scores.

[0120] Further, the device further comprises:

[0121] The screening module is configured to screen the image detection result and the point cloud detection result to be obtained to obtain the image detection result and the point cloud detection result.

[0122] Further, the screening module comprises:

[0123] The comparison unit is configured to compare the image detection score of the image detection result to be obtained with a set image detection threshold value, and if the image detection score is greater than the image detection threshold value, the image detection result is obtained.

[0124] The comparison unit is further configured to compare the point cloud detection score of the point cloud detection result to be obtained with a set point cloud detection threshold value, and if the point cloud detection score is greater than the point cloud detection threshold value, the point cloud detection result is obtained.

[0125] Further, the association module comprises:

[0126] The first judging unit is configured to judge whether the first label of the image detection result and the second label of the point cloud detection result are the same.

[0127] The obtaining unit is configured to obtain the first projection center of the image detection result and the depth of the image detection result if the first judging unit judges that the first label and the second label are the same.

[0128] The first calculating unit is configured to calculate the second projection center of the point cloud detection result and the depth of the point cloud detection result.

[0129] The second calculating unit is configured to calculate the distance between the first projection center and the second projection center and the difference value between the depth of the image detection result and the depth of the point cloud detection result.

[0130] The association unit is configured to determine whether to match and associate the image detection result and the point cloud detection result according to the distance and the difference value.

[0131] Further, the first calculation unit comprises:

[0132] a projection center sub-module, configured to determine the second projection center according to the first projection center and the bias of the point cloud detection result relative to the second projection center;

[0133] a depth sub-module, configured to determine the depth of the point cloud detection result according to the depth information of the image detection result and the depth bias of the point cloud detection result.

[0134] Further, the fusion module comprises:

[0135] a second judgment unit, configured to judge the confidence score;

[0136] a depth unit, configured to, if the judgment result of the judgment unit is that the confidence score is greater than a set confidence threshold, determine the depth of the fusion detection result according to the depth of the point cloud detection result and the confidence score;

[0137] a projection center unit, configured to determine the projection center of the fusion detection result according to the projection center of the point cloud detection result and the confidence score;

[0138] a fusion unit, configured to, if the judgment result of the judgment unit is less than or equal to a set confidence threshold, take the image detection result as the fusion detection result.

[0139] In the embodiment, the image detection result and the point cloud detection result are obtained, wherein the point cloud detection result is obtained by fusing the image and the point cloud data; the image detection result and the point cloud detection result belonging to the same object are matched and associated; a plurality of point cloud detection results are input into a neural network model to obtain a plurality of confidence scores corresponding to the point cloud detection result; and the fusion detection result is obtained according to the image detection result, the point cloud detection result and the confidence score. That is, the image features are used to enhance the millimeter wave radar features in the present application, the enhanced millimeter wave radar features are used to realize image-radar association at the detection level, and the enhanced millimeter wave radar depth is used to improve the depth estimation and the projection center of the multi-camera vision, so as to obtain the fusion detection result, which integrates the advantages of the image and the point cloud data, thereby realizing more accurate and comprehensive three-dimensional object detection capability and improving the perception accuracy and reliability of the automatic driving system. For the above-mentioned device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the related parts can be referred to the part of the method embodiment.

[0140] It is readily apparent to those skilled in the art that any combination of the above-described embodiments is possible and, therefore, any combination of the above-described embodiments is an embodiment of the present application, but due to the limitation of the length of the specification, the specification will not be described here in detail.

[0141] Although preferred embodiments of the application have been described herein, additional changes and modifications can be suggested to one skilled in the art, particularly in light of the essential novel teachings set forth herein. It is intended, therefore, that the present application be construed as including all such changes and modifications as fall within the scope of the claimed application. The foregoing embodiments are presented by way of example only and are not intended to limit the scope of the present application, which is set forth in the claims appended hereto. Equivalent alternatives to the embodiments described herein which do not depart from the spirit and scope of the present application are within the scope of the claims.

Claims

1. An image and point cloud fusion method, characterized in that, The method comprises: obtaining image detection results and point cloud detection results, wherein the point cloud detection results are obtained by fusing image and point cloud data; matching and associating the image detection results and the point cloud detection results belonging to the same object; inputting a plurality of the point cloud detection results into a neural network model to obtain a plurality of confidence scores corresponding to the point cloud detection results; obtaining fusion detection results according to the image detection results, the point cloud detection results and the confidence scores; the matching and associating the image detection results and the point cloud detection results belonging to the same object comprises: judging whether a first label of the image detection result and a second label of the point cloud detection result are the same; if the first label and the second label are the same, obtaining a first projection center of the image detection result and a depth of the image detection result; calculating a second projection center of the point cloud detection result and a depth of the point cloud detection result; calculating a distance between the first projection center and the second projection center and a difference value of the depth of the image detection result and the depth of the point cloud detection result; determining whether to match and associate the image detection result and the point cloud detection result according to the distance and the difference value.

2. The method of claim 1, wherein, Before the obtaining image detection results and point cloud detection results, the method further comprises: screening the to-be-obtained image detection results and the to-be-obtained point cloud detection results to obtain the image detection results and the point cloud detection results.

3. The method of claim 2, wherein, The screening the to-be-obtained image detection results and the to-be-obtained point cloud detection results to obtain the image detection results and the point cloud detection results comprises: comparing an image detection score of the to-be-obtained image detection result with a set image detection threshold value, and if the image detection score is greater than the image detection threshold value, obtaining the image detection result; comparing a point cloud detection score of the to-be-obtained point cloud detection result with a set point cloud detection threshold value, and if the point cloud detection score is greater than the point cloud detection threshold value, obtaining the point cloud detection result.

4. The method of claim 1, wherein, The calculating the second projection center of the point cloud detection result and the depth of the point cloud detection result comprises: determining the second projection center according to the first projection center and the bias of the point cloud detection result relative to the second projection center; determining the depth of the point cloud detection result according to the depth information of the image detection result and the depth bias of the point cloud detection result.

5. The method of claim 1, wherein, The obtaining fusion detection results according to the image detection results, the point cloud detection results and the confidence scores comprises: judging the confidence score; if the confidence score is greater than a set confidence threshold value, determining the depth of the fusion detection result according to the depth of the point cloud detection result and the confidence score; determining the projection center of the fusion detection result according to the projection center of the point cloud detection result and the confidence score; if the confidence score is less than or equal to the set confidence threshold value, taking the image detection result as the fusion detection result.

6. An image and point cloud fusion device, characterized in that, The method comprises: The acquisition module is configured to acquire an image detection result and a point cloud detection result, wherein the point cloud detection result is obtained by fusing image and point cloud data; The association module is configured to match and associate the image detection result and the point cloud detection result belonging to the same object; The confidence module is configured to input a plurality of the point cloud detection results into a neural network model to obtain a plurality of confidence scores corresponding to the point cloud detection results; The fusion module is configured to obtain a fusion detection result according to the image detection result, the point cloud detection result and the confidence scores; The association module comprises: The first judging unit is configured to judge whether a first label of the image detection result and a second label of the point cloud detection result are the same; The obtaining unit is configured to, if the first judging unit judges that the first label and the second label are the same, obtain a first projection center of the image detection result and a depth of the image detection result; The first calculating unit is configured to calculate a second projection center of the point cloud detection result and a depth of the point cloud detection result; The second calculating unit is configured to calculate a distance between the first projection center and the second projection center and a difference value of the depth of the image detection result and the depth of the point cloud detection result; The association unit is configured to determine whether to match and associate the image detection result and the point cloud detection result according to the distance and the difference value.

7. The apparatus of claim 6, wherein, The device further comprises: The screening module is configured to screen a to-be-acquired image detection result and a to-be-acquired point cloud detection result to obtain the image detection result and the point cloud detection result.

8. The apparatus of claim 7, wherein, The screening module comprises: The comparison unit is configured to compare an image detection score of the to-be-acquired image detection result with a set image detection threshold value, and if the image detection score is greater than the image detection threshold value, the image detection result is obtained; The comparison unit is further configured to compare a point cloud detection score of the to-be-acquired point cloud detection result with a set point cloud detection threshold value, and if the point cloud detection score is greater than the point cloud detection threshold value, the point cloud detection result is obtained.

9. The apparatus of claim 6, wherein, The first calculating unit comprises: The projection center submodule is configured to determine the second projection center according to the first projection center and the bias of the point cloud detection result relative to the second projection center; The depth submodule is configured to determine the depth of the point cloud detection result according to the depth information of the image detection result and the depth bias of the point cloud detection result.

10. The apparatus of claim 6, wherein, The fusion module comprises: The second judging unit is configured to judge the confidence scores; The depth unit is configured to, if the judging result of the judging unit is that the confidence scores are greater than a set confidence threshold value, determine the depth of the fusion detection result according to the depth of the point cloud detection result and the confidence scores; The projection center unit is configured to determine the projection center of the fusion detection result according to the projection center of the point cloud detection result and the confidence scores; The fusion unit is configured to, if the judging result of the judging unit is less than or equal to the set confidence threshold value, take the image detection result as the fusion detection result.

Images