A camera and lidar fusion target detection method and device

Abstract

The present invention discloses a target detection method that integrates camera and laser radar, comprising: using the YOLOV5 model to perform target detection on color image data of a target scene to obtain a set of visual detection results; using the laser radar to obtain point cloud data of the target scene, and detecting suspected targets through a clustering algorithm to obtain a set of suspected target detection results; determining the bounding box Box of the first target with the highest target confidence from the set of visual detection results. o , and determine the suspected target detection result set L and Box o Matching bounding box Box′ l ; Get Box′ l The corresponding point cloud data is classified and the first confidence level is obtained; according to the first confidence level, Box o The corresponding confidence, Box′ l and Box o Determine the location and classification results of the target, and according to Box′ l The corresponding point cloud data determines the distance information of the target. The present invention can effectively improve the problems of camera misdetection and low detection confidence due to poor lighting.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of target detection, and particularly relates to a camera and laser radar fusion target detection method and device. BACKGROUND

[0002] In automatic driving, the environment perception system is the biggest guarantee for the safety of vehicle driving, and provides a basis for high-level tasks such as path planning and behavior decision of the subsequent vehicle. Laser radar point cloud data can provide depth information of the target, and has low sensitivity to weather and light. However, the point cloud data has the characteristics of disorder, sparseness and non-structure, which increases the difficulty of laser radar application. The camera has low price, can obtain color information, and the development technology is mature. However, the camera is greatly affected by light, the detection effect is not good, and three-dimensional information cannot be obtained. Since a single sensor cannot meet the demand of automatic driving for environment perception, the camera and the laser radar can be fused to compensate for the shortcomings of the two.

[0003] Silva et al. [1] used a geometric model to spatially align the laser radar point cloud and the color image, then used Gaussian process regression for depth completion, so that the two sensors had the same resolution, and finally verified that the algorithm had obvious auxiliary effect on the subsequent perception step by using free space detection. However, the depth completion algorithm used in the paper is only guided by the image and too dependent on the image, which will cause unknown pixel points with similar colors and large real depth differences to also be completed to similar depth values. Premebida et al. [2] combined laser radar and RGB camera for pedestrian detection. First, a dense depth map is generated by laser radar point cloud, and then DPM is used to detect pedestrians on the laser radar dense depth map and color image respectively, and then SVM is used for re-scoring, and finally the fusion detection is realized. However, the generation of the dense depth map only relies on the laser radar data, and the guiding information is too single, resulting in large error of the completed value. Kang et al. [3] designed a convolutional neural network composed of independent unary classifiers and fusion CNN, which fused the laser radar three-dimensional point cloud and the color image, and realized high-precision target detection. However, a large amount of calculation is required, and real-time performance cannot be guaranteed.

[0004] In summary, how to accurately perceive and identify targets in automatic driving, especially in high-speed driving, solve the real-time problem and precision problem of detecting targets, and quickly and accurately realize target detection is a challenging problem. SUMMARY

[0005] In order to solve the above problems in the prior art, the application provides a camera and laser radar fusion target detection method and device. The technical problem to be solved by the application is solved by the following technical scheme.

[0006] In a first aspect, the present application provides a camera and lidar fusion target detection method, comprising:

[0007] YOLOV5 model is used for target detection on color image data of a target scene, to obtain a visual detection result set O{Box1, Box2,..., Box n}, which includes target confidence of each first target, Box1, Box2,..., Box n represent the first, second,..., n first target boundary boxes in the visual detection result, respectively;

[0008] Lidar is used to obtain point cloud data of the target scene, and a clustering algorithm is used to detect suspected targets from the point cloud data, to obtain a suspected target detection result set L{Box'1, Box'2,..., Box'm m}, Box'1, Box'2,..., Box'm m represent the first, second,..., m suspected target boundary boxes in the suspected target detection result, respectively;

[0009] The boundary box Box n of the first target with the highest target confidence is determined from the visual detection result set O{Box1, Box2,..., Box o , and the boundary box Box' m that matches Box o is determined from the suspected target detection result set L{Box'1, Box'2,..., Box'm l};

[0010] The point cloud data corresponding to the boundary box Box' l is obtained and input into a PointNet network for classification, to obtain a first confidence;

[0011] The target position and classification result are determined according to the first confidence, the confidence corresponding to the boundary box Box o , the boundary box Box' l , and the boundary box Box o , and the distance information of the target is determined according to the point cloud data corresponding to the boundary box Box' l .

[0012] In an embodiment of the present application, the boundary box Box n of the first target with the highest target confidence is determined from the visual detection result set O{Box1, Box2,..., Box o ., and determine the suspected target detection result set L{Box′1, Box′2, ..., Box′ m} and Box o The matching bounding box Box′1 includes:

[0013] From the visual detection result set O{Box1, Box2, ..., Box n}Determine the bounding box Box of the first target with the largest target confidence o ;

[0014] Calculate the bounding box Box separately o and the suspected target detection result set L{Box′1, Box′2, ..., Box′ m The intersection-over-union ratio of the bounding boxes of each suspected target in};

[0015] Determine the maximum value of the intersection-over-union ratio Iou max , and the maximum value Iou max In the suspected target detection result set L{Box′1, Box′2, ..., Box′ m} in the corresponding bounding box is determined to be the same as Box o Matching bounding box Box′ l .

[0016] In one embodiment of the present invention, the bounding box Box′ is obtained l The corresponding point cloud data is input into the PointNet network for classification to obtain the first confidence level, including:

[0017] When the maximum value Iou max When it is greater than or equal to the first threshold, the bounding box Box′ is obtained. l The corresponding point cloud data is input into the PointNet network for classification. After obtaining the first confidence, the visual detection result set O{Box1, Box2, ..., Box n} to delete the bounding box Box o , from the suspected target detection result set L{Box′1, Box′2, ..., Box′ m}Delete Box′ l ;

[0018] When the maximum value Iou max When it is less than the first threshold and greater than or equal to the second threshold, the bounding box Box is obtained. o The corresponding point cloud data is input into the PointNet network for classification. After obtaining the first confidence, the visual detection result set O{Box1, Box2, ..., Box ndelete the bounding box Box o .

[0019] In an embodiment of the present application, the position and classification result of the target are determined according to the first confidence, the bounding box Box o the corresponding confidence, the bounding box Box' l and the bounding box Box o The step of determining the distance information of the target according to the corresponding point cloud data of the bounding box Box' l includes:

[0020] The classification result of the target is obtained by fusing the first confidence and the confidence corresponding to the bounding box Box o .

[0021] The position of the target is obtained by fusing the bounding box Box' l and the bounding box Box o .

[0022] The distance information of the corresponding point cloud data of the bounding box Box' l is obtained by using the laser radar.

[0023] In an embodiment of the present application, the classification result of the target is obtained by fusing the first confidence and the confidence corresponding to the bounding box Box o . The step includes:

[0024] The first confidence and the confidence corresponding to the bounding box Box o are weighted and averaged according to a first preset weight and a second preset weight:

[0025]

[0026] In the formula, f1 represents the first preset weight, f2 represents the second preset weight, x1 represents the first confidence, and x2 represents the confidence corresponding to the bounding box Box o .

[0027] In an embodiment of the present application, the bounding box of each first target in the set of visual detection results is a rectangular bounding box, and the bounding box of each suspected target in the set of suspected target detection results is a rectangular bounding box.

[0028] The position of the target is obtained by fusing the bounding box Box' l and the bounding box Box o . The step includes:

[0029] The bounding box Box' la first coordinate of four vertices of the bounding box Box o a second coordinate of four vertices of the bounding box Box

[0030] determining a maximum value of the horizontal coordinate, a minimum value of the horizontal coordinate, a maximum value of the vertical coordinate and a minimum value of the vertical coordinate according to the first coordinate and the second coordinate;

[0031] generating a fused bounding box by taking the maximum value of the horizontal coordinate, the minimum value of the horizontal coordinate, the maximum value of the vertical coordinate and the minimum value of the vertical coordinate as vertices.

[0032] In a second aspect, the present application provides a camera and laser radar fused target detection device, comprising:

[0033] a visual detection module, configured to perform target detection on color image data of a target scene by using a YOLOV5 model to obtain a visual detection result set O{Box1, Box2,..., Box n}, wherein the visual detection result set includes a target confidence of each first target, Box1, Box2,..., Box n representing a bounding box of the 1st, 2nd,..., nth first target in the visual detection result, respectively;

[0034] a target recognition module, configured to obtain a point cloud data of a target scene by using a laser radar, and perform detection on the point cloud data by using a clustering algorithm to obtain a suspected target detection result set L{Box′1, Box′2,..., Box′ m}, Box′1, Box′2,..., Box′ m representing a bounding box of the 1st, 2nd,..., mth suspected target in the suspected target detection result, respectively;

[0035] a matching module, configured to determine a bounding box Box n of a first target with the maximum target confidence from the visual detection result set O{Box1, Box2,..., Box o , and determine a bounding box Box′ m matching Box o from the suspected target detection result set L{Box′1, Box′2,..., Box′ l};

[0036] a classification module, configured to obtain corresponding point cloud data of the bounding box Box′ l and input the point cloud data into a PointNet network for classification to obtain a first confidence;

[0037] The detection result determination module is used to determine the detection result based on the first confidence level and the bounding box Box o The corresponding confidence, the bounding box Box′ l and the bounding box Box o Determine the location and classification results of the target and classify it according to the bounding box Box′ l The corresponding point cloud data determines the distance information of the target.

[0038] Compared with the prior art, the present invention has the following beneficial effects:

[0039] This invention provides a method and device for target detection that integrates camera and lidar. By employing a fusion strategy that prioritizes visual detection results and supplements them with lidar detection results, this method effectively mitigates issues such as camera misdetection and low detection confidence due to poor lighting conditions. Furthermore, visual detection overcomes the inaccurate obstacle clustering inherent in lidar. This fusion and matching strategy ensures a certain degree of redundancy in the information sensed by each sensor, while utilizing point cloud ranging to obtain accurate distance information.

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 This is a flow chart of a target detection method that integrates camera and lidar according to an embodiment of the present invention;

[0042] Figure 2 Schematic diagram of a laser radar provided by an embodiment of the present invention performing suspected target recognition;

[0043] Figure 3 is another flow chart of the target detection method for camera and lidar fusion provided by an embodiment of the present invention;

[0044] Figure 4 is a schematic diagram of the fusion of bounding boxes provided by an embodiment of the present invention;

[0045] Figure 5 is a schematic diagram of target detection results provided by an embodiment of the present invention;

[0046] Figure 6 It is a structural diagram of a target detection device that integrates a camera and a lidar provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0047] The present invention will be further described in detail below with reference to specific examples, but the embodiments of the present invention are not limited thereto.

[0048] Figure 1This is a flow chart of a target detection method that integrates camera and laser radar according to an embodiment of the present invention. Figure 1 As shown, an embodiment of the present invention provides a target detection method integrating a camera and a laser radar, including:

[0049] S1. Use the YOLOV5 model to perform target detection on the color image data of the target scene and obtain the visual detection result set O{Box1, Box2, ..., Box n}, the visual detection result set includes the target confidence of each first target, Box1, Box2, ..., Box n Respectively represent the bounding boxes of the 1st, 2nd, ..., nth first targets in the visual detection results;

[0050] S2. Use the laser radar to obtain the point cloud data of the target scene, and use the clustering algorithm to detect suspected targets on the point cloud data to obtain the suspected target detection result set L{Box′1, Box′2, ..., Box′ m}, Box′1, Box′2,..., Box′ m Respectively represent the bounding boxes of the 1st, 2nd…, mth suspected targets in the suspected target detection results;

[0051] S3, from the visual detection result set O{Box1, Box2, ..., Box n}Determine the bounding box Box of the first target with the largest target confidence o , and determine the suspected target detection result set L{Box′1, Box′2, ..., Box′ m} and Box o Matching bounding box Box′ l ;

[0052] S4. Get the bounding box Box′ l The corresponding point cloud data is input into the PointNet network for classification to obtain the first confidence level;

[0053] S5. According to the first confidence level, bounding box Box o Corresponding confidence, bounding box Box′ l and bounding box Box o Determine the location and classification results of the target and classify it according to the bounding box Box′ l The corresponding point cloud data determines the distance information of the target.

[0054] In this embodiment, the lightweight YOLOV5 model is first used to perform target detection on color image data, wherein the color image data can be obtained by shooting the target scene with a camera, and the visual detection result set obtained by the detection is O{Box1, Box2, ..., Box n}, Box1, Box2, ..., Box n They represent the bounding boxes of the 1st, 2nd, ..., nth first targets in the visual detection results. Each detected first target in this set has four attributes, namely: target height h o , target width w o , target category class, and target confidence conf. It should be noted that when using the YOLOV5 model for target detection, the output confidence evaluation threshold of the visually detected target can be lowered to effectively output the target bounding boxes with low confidence in the results. This allows the low-confidence visual detection targets to be fused with the suspected LiDAR targets in the subsequent matching process, effectively improving the false detection phenomenon caused by a single sensor (camera).

[0055] In the above step S2, the laser radar is used to obtain the point cloud data of the target scene and detect suspected targets. Figure 2 FIG is a schematic diagram of a laser radar provided by an embodiment of the present invention for identifying a suspected target. Figure 2 As shown, in this embodiment, the bounding box of the detected suspected target can be projected into the image pixel coordinate system to obtain the suspected target detection result set L{Box′1, Box′2, ..., Box′ m}, Box′1, Box′2,..., Box′ m They represent the bounding boxes of the 1st, 2nd, ..., mth suspected targets in the suspected target detection results. Each detected suspected target in this set has two attributes: height h l and width w l .

[0056] Furthermore, considering that after lowering the confidence evaluation threshold, the first target with low visual detection confidence is not filtered out, there may be a situation where there are multiple bounding boxes on a target. Therefore, it is most reasonable to traverse the bounding boxes in the visual detection result set from high to low confidence. Specifically, if the visual detection result set O is not empty, then Box1, Box2, ..., Box n Sorting, determine the bounding box Box of the first target with the largest target confidence from the visual detection result set O o Next, calculate the bounding box Box o with Box′1, Box′2, ..., Box′ mThe intersection-and-union ratio is matched by judging the intersection-and-union ratio. If the bounding box Box′ in the suspected target detection result set L l and bounding box Box o Match, then the bounding box Box′ l The point cloud data is classified and the target detection result is obtained. On the contrary, if the bounding box Box′ l and bounding box Box o If the bounding box Box′ cannot be determined, the visual detection result is used as the main one. l and bounding box Box o If there is a match, extract the bounding box Box o The inner point cloud is classified and the target detection result is obtained.

[0057] Figure 3 This is another flow chart of the target detection method of camera and laser radar fusion provided by the embodiment of the present invention. Figure 3 As shown, in the above step S3, from the visual detection result set O{Box1, Box2, ..., Box n}Determine the bounding box Box of the first target with the largest target confidence o , and determine the suspected target detection result set L{Box′1, Box′2, ..., Box′ m} and Box o The matching bounding box Box′1 includes:

[0058] S301, from the visual detection result set O{Box1, Box2, ..., Box n}Determine the bounding box Box of the first target with the largest target confidence o ;

[0059] S302, calculate the bounding box Box respectively o and the suspected target detection result set L{Box′1,Box′2,...,Box′ m The intersection-over-union ratio of the bounding boxes of each suspected target in};

[0060] S303, determine the maximum value of the intersection-over-union ratio Iou max , and the maximum value Iou max In the suspected target detection result set L{Box′1, Box′2, ..., Box′ m} in the corresponding bounding box is determined to be the same as Box o Matching bounding box Box′ l .

[0061] Specifically, from the visual detection result set O{Box1, Box2, ..., Boxn} to extract the first target with the maximum target confidence o , calculate the intersection over union with each bounding box of the suspected target in the suspected target detection result set L, and calculate the maximum value Iou max .

[0062] In the above step S4, the boundary box Box′ l corresponding point cloud data is obtained and input into the PointNet network for classification to obtain the first confidence.

[0063] When the maximum value Iou max is greater than or equal to the first threshold, the boundary box Box′ l corresponding point cloud data is obtained and input into the PointNet network for classification to obtain the first confidence, and the boundary box Box n is deleted from the visual detection result set O{Box1, Box2,..., Box o}, and the boundary box Box′ m is deleted from the suspected target detection result set L{Box′1, Box′2,..., Box′ l};

[0064] When the maximum value Iou max is less than the first threshold and greater than or equal to the second threshold, the boundary box Box′ o corresponding point cloud data is obtained and input into the PointNet network for classification to obtain the first confidence, and the boundary box Box n is deleted from the visual detection result set O{Box1, Box2,..., Box o}. It should be understood that this situation is due to inaccurate clustering of point cloud data, resulting in adjacent targets being clustered into one suspected target due to being too close.

[0065] In addition, when the maximum value Iou max is less than the second threshold, the target confidence of the visual detection result needs to be judged. If the target confidence is greater than or equal to the third threshold, it indicates that the visual detection result is reliable, otherwise it is a false detection, and the boundary box Box n is deleted from the visual detection result set O{Box1, Box2,..., Box o}. It should be noted that this situation indicates that the laser radar does not find the suspected target, or is caused by visual error.

[0066] In the above step S5, according to the first confidence, the confidence corresponding to the boundary box Box′ o , the boundary box Box′ l and the boundary box Boxo determining the position and classification result of the target, and determining distance information of the target according to the bounding box Box' l The step of determining the distance information of the target according to the corresponding point cloud data comprises:

[0067] S501, fusing the first confidence and the bounding box Box o to obtain the classification result of the target;

[0068] S502, fusing the bounding box Box' l and the bounding box Box o to obtain the position of the target;

[0069] S503, obtaining the distance information of the corresponding point cloud data by using the laser radar to obtain the bounding box Box' l

[0070] In the embodiment, when fusing the first confidence and the bounding box Box o corresponding confidence, the first preset weight and the second preset weight can be used for weighted average of the two:

[0071]

[0072] In the formula, f1 represents the first preset weight, f2 represents the second preset weight, x1 represents the confidence corresponding to the bounding box Box o , and x2 represents the first confidence. In order to ensure the accuracy of the low-confidence visual detection box, the embodiment gives a high weight to the confidence of the suspected target output by the point cloud classification network, and exemplarily, f1=0.2 and f2=0.8.

[0073] Further, the bounding box of each first target in the visual detection result set is a rectangular bounding box, and the bounding box of each suspected target in the suspected target detection result set is a rectangular bounding box. Figure 4 is a fusion schematic diagram of the bounding box provided by the embodiment of the application. As shown in Figure 4 in step S502, the bounding box Box' l and the bounding box Box o are fused to obtain the position of the target, which comprises:

[0074] obtaining the first coordinates of the four vertices of the bounding box Box' l in the image pixel coordinate system and the second coordinates of the four vertices of the bounding box Box o in the image pixel coordinate system;

[0075] determining the maximum value of the horizontal coordinate, the minimum value of the horizontal coordinate, the maximum value of the vertical coordinate, and the minimum value of the vertical coordinate according to the first coordinates and the second coordinates.​

[0076] The maximum value of the horizontal coordinate, the minimum value of the horizontal coordinate, the maximum value of the vertical coordinate, and the minimum value of the vertical coordinate are used as vertices to generate the fused bounding box.

[0077] The accurate positioning of the bounding box is the key to target detection. It can clearly reflect the position and size of the target vehicle. Therefore, obtaining accurate bounding box information is crucial for vehicle detection on the road. This embodiment combines color images with point cloud technology to obtain a more refined bounding box, such as Figure 4 As shown in the figure, frame 1 represents the bounding box detected from the color image data, frame 2 represents the bounding box of the suspected target identified by the lidar, and frame 3 is the more refined bounding box formed by fusing the two. It should be understood that both bounding boxes are considered valid detection areas. Therefore, these two bounding boxes are extended outward to form a new rectangular area to determine the final target detection result.

[0078] Figure 5 is a schematic diagram of the target detection result provided by the embodiment of the present invention. Figure 5 As shown in the figure, the distance between the current driving vehicle and the preceding vehicle can be determined by using the position information provided by the bounding box and the distance information provided by the LiDAR. The image and point cloud obstacle detection results are fused, and the point cloud obstacle clustering bounding boxes that meet the rules are retained. The bounding box point cloud data is extracted to obtain the distance information for all positions of the target vehicle. This data is averaged to obtain the distance between the LiDAR and the preceding vehicle. The distance between the LiDAR and the vehicle's front end is then subtracted to obtain the final inter-vehicle distance.

[0079] Target detection is performed using the target detection method provided by the present invention and the YOLOV5 model, and the results are shown in Table 1.

[0080] Table 1

[0081]

[0082] As can be seen from Table 1, the target detection method of the camera and lidar fusion provided by the present invention has improved detection accuracy.

[0083] Figure 6 Schematic diagram of the structure of the target detection device fused with camera and laser radar provided by the embodiment of the present invention. Figure 6 As shown, an embodiment of the present invention further provides a target detection device integrating a camera and a laser radar, comprising:

[0084] The visual detection module 610 is used to perform target detection on the color image data of the target scene using the YOLOV5 model to obtain a visual detection result set O{Box1, Box2, ..., Box n}, Box1, Box2,..., Box n respectively represent the bounding boxes of the 1st, 2nd,..., nth first target in the visual detection result;

[0085] The target recognition module 620 is configured to acquire point cloud data of a target scene by using a laser radar, and detect a suspected target in the point cloud data by using a clustering algorithm to obtain a suspected target detection result set L{Box'1, Box'2,..., Box'm} m}, Box'1, Box'2,..., Box'm m respectively represent the bounding boxes of the 1st, 2nd,..., mth suspected target in the suspected target detection result;

[0086] The matching module 630 is configured to determine the bounding box Box n of the first target with the highest target confidence from the visual detection result set O{Box1, Box2,..., Box o , and determine the bounding box Box' m matching the bounding box Box o in the suspected target detection result set L{Box'1, Box'2,..., Box'm} l ;

[0087] The classification module 640 is configured to acquire the point cloud data corresponding to the bounding box Box' l and input the point cloud data into a PointNet network for classification to obtain a first confidence;

[0088] The detection result determination module 650 is configured to determine the position and classification result of the target according to the first confidence, the confidence corresponding to the bounding box Box' o , the bounding box Box' l and the bounding box Box o , and determine the distance information of the target according to the point cloud data corresponding to the bounding box Box' l .

[0089] It can be known from the above embodiments that the present application has the following beneficial effects:

[0090] The present application provides a camera and laser radar fusion target detection method and device, which can effectively improve the problems of camera false detection and low detection confidence caused by poor light by using a fusion strategy mainly based on visual detection results and supplemented by laser radar detection results. Meanwhile, the visual detection results also overcome the disadvantage of inaccurate laser radar obstacle clustering. The present application uses the above fusion matching strategy, so that the information perceived by each sensor has a certain redundancy, and the accurate distance information is obtained by using point cloud ranging.

[0091] In the description of the application, the terms "first", "second", "third", etc. are used only to describe different embodiments and do not imply or suggest relative importance or a specific number of the technical features indicated. Therefore, the features defined with "first", "second", etc. can explicitly or implicitly include one or more of the features. In the description of the application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0092] In the description of the specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. In the specification, the illustrative description of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the specification.

[0093] Although the application is described herein in conjunction with various embodiments, other variations of the disclosed embodiments can be understood and effected by those skilled in the art in the practice of the claimed application, by viewing the drawings, disclosure, and appended claims.

[0094] The above is a further detailed description of the application in conjunction with specific preferred embodiments, and cannot be considered as limiting the specific implementation of the application to these descriptions. For those skilled in the art, without departing from the concept of the application, a number of simple deductions or substitutions can be made, which should be considered as falling within the protection scope of the application.

Claims

1. A target detection method integrating camera and laser radar, characterized in that: include: The YOLOV5 model is used to perform target detection on the color image data of the target scene, and a set of visual detection results O{Box1, Box2, ..., Box n }, the visual detection result set includes the target confidence of each first target, Box1, Box2, ..., Box n Respectively represent the bounding boxes of the 1st, 2nd, ..., nth first targets in the visual detection results; The laser radar is used to obtain the point cloud data of the target scene, and the clustering algorithm is used to detect the suspected targets in the point cloud data to obtain the suspected target detection result set L{Box′1, Box′2, ..., Box′ m }, Box′1, Box′2,..., Box′ m Respectively represent the bounding boxes of the 1st, 2nd…, mth suspected targets in the suspected target detection results; From the visual detection result set O{Box1, Box2, ..., Box n }Determine the bounding box Box of the first target with the largest target confidence o , and determine the suspected target detection result set L{Box′1, Box′2, ..., Box′ m } and Box o Matching bounding box Box′ l ; Get the bounding box Box′ l The corresponding point cloud data is input into the PointNet network for classification to obtain the first confidence level; According to the first confidence, the bounding box Box o The corresponding confidence, the bounding box Box′ l and the bounding box Box o Determine the location and classification results of the target and classify it according to the bounding box Box′ l The corresponding point cloud data determines the distance information of the target.

2. The target detection method of camera and laser radar fusion according to claim 1, characterized in that: From the visual detection result set O{Box1, Box2, ..., Box n }Determine the bounding box Box of the first target with the largest target confidence o , and determine the suspected target detection result set L{Box′1, Box′2, ..., Box′ m } and Box o Matching bounding box Box′ l ,include: From the visual detection result set O{Box1, Box2, ..., Box n }Determine the bounding box Box of the first target with the largest target confidence o ; Calculate the bounding box Box separately o and the suspected target detection result set L{Box′1, Box′2, ..., Box′ m The intersection-over-union ratio of the bounding boxes of each suspected target in}; Determine the maximum value of the intersection-over-union ratio Iou max , and the maximum value Iou max In the suspected target detection result set L{Box′1, Box′2, ..., Box′ m } in the corresponding bounding box is determined to be the same as Box o Matching bounding box Box′ l .

3. The target detection method of camera and laser radar fusion according to claim 2, characterized in that: Get the bounding box Box′ l The corresponding point cloud data is input into the PointNet network for classification to obtain the first confidence level, including: When the maximum value Iou max When it is greater than or equal to the first threshold, the bounding box Box′ is obtained. l The corresponding point cloud data is input into the PointNet network for classification. After obtaining the first confidence, the visual detection result set O{Box1, Box2, ..., Box n } to delete the bounding box Box o , from the suspected target detection result set L{Box′1, Box′2, ..., Box′ m }Delete Box′ l ; When the maximum value Iou max When it is less than the first threshold and greater than or equal to the second threshold, the bounding box Box is obtained. o The corresponding point cloud data is input into the PointNet network for classification. After obtaining the first confidence, the visual detection result set O{Box1, Box2, ..., Box n } to delete the bounding box Box o .

4. The target detection method of camera and laser radar fusion according to claim 2, characterized in that: According to the first confidence, the bounding box Box o The corresponding confidence, the bounding box Box′ l and the bounding box Box o Determine the location and classification results of the target and classify it according to the bounding box Box′ l The steps of determining the distance information of the target based on the corresponding point cloud data include: By fusing the first confidence and the bounding box Box o The corresponding confidence levels are fused to obtain the classification results of the target; By fusing the bounding box Box′ l and the bounding box Box o , get the location of the target; Use lidar to obtain the bounding box Box′ l The distance information of the corresponding point cloud data.

5. The target detection method of camera and laser radar fusion according to claim 4, characterized in that: By fusing the first confidence and the bounding box Box o The steps of fusing the corresponding confidence levels to obtain the classification results of the target include: According to the first preset weight and the second preset weight, the first confidence and the bounding box Box o The corresponding confidence levels are weighted averaged: Where f1 represents the first preset weight, f2 represents the second preset weight, x1 represents the first confidence, and x2 represents the bounding box Box o The corresponding confidence level.

6. The target detection method of camera and laser radar fusion according to claim 4, characterized in that: The bounding box of each first target in the visual detection result set is a rectangular bounding box, and the bounding box of each suspected target in the suspected target detection result set is a rectangular bounding box; By fusing the bounding box Box′ l and the bounding box Box o , the steps of obtaining the location of the target include: Get the bounding box Box′ l The first coordinates of the four vertices in the image pixel coordinate system, and the bounding box Box o The second coordinates of the four vertices in the image pixel coordinate system; Determining a maximum value of the horizontal coordinate, a minimum value of the horizontal coordinate, a maximum value of the vertical coordinate, and a minimum value of the vertical coordinate based on the first coordinate and the second coordinate; The maximum value of the horizontal coordinate, the minimum value of the horizontal coordinate, the maximum value of the vertical coordinate, and the minimum value of the vertical coordinate are used as vertices to generate a fused bounding box.

7. A target detection device integrating a camera and a laser radar, characterized in that: include: The visual detection module is used to perform target detection on the color image data of the target scene using the YOLOV5 model, and obtain the visual detection result set O{Box1, Box2, ..., Box n }, the visual detection result set includes the target confidence of each first target, Box1, Box2, ..., Box n Respectively represent the bounding boxes of the 1st, 2nd, ..., nth first targets in the visual detection results; The target recognition module is used to obtain the point cloud data of the target scene using the laser radar, and detect the suspected targets on the point cloud data through the clustering algorithm to obtain the suspected target detection result set L{Box′1, Box′2, ..., Box′ m }, Box′1, Box′2,..., Box′ m Respectively represent the bounding boxes of the 1st, 2nd…, mth suspected targets in the suspected target detection results; A matching module is used to select the visual detection result set O{Box1, Box2, ..., Box n }Determine the bounding box Box of the first target with the largest target confidence o , and determine the suspected target detection result set L{Box′1, Box′2, ..., Box′ m } and Box o Matching bounding box Box′ l ; Classification module, used to obtain the bounding box Box′ l The corresponding point cloud data is input into the PointNet network for classification to obtain the first confidence level; The detection result determination module is used to determine the detection result based on the first confidence level and the bounding box Box o The corresponding confidence, the bounding box Box′ l and the bounding box Box o Determine the location and classification results of the target and classify it according to the bounding box Box′ l The corresponding point cloud data determines the distance information of the target.

Images