Non-maximum suppression based laser point cloud and image asymmetric fusion target detection method and device and storage medium

Abstract

The application provides a laser point cloud and image asymmetric fusion target detection method based on non-maximum suppression, a single-stage image two-dimensional target detection network model, dynamic adjustment of a confidence of a single-stage image two-dimensional target detection network model output prediction frame and an IoU filtering threshold according to original data of a laser radar laser point cloud, non-maximum suppression, and then extraction of a real target detection frame from multiple output prediction frames and filtering of non-real target detection frames. The application can improve the accuracy of two-dimensional target detection, avoid inflation of a real target into multiple virtual targets, thereby reducing the false detection rate, and also avoid aggregation of multiple real targets into a single virtual target.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method, apparatus and storage medium for target detection based on nonmaximum suppression-based asymmetric fusion of laser point cloud and image. Background Technology

[0002] Image 2D object detection is an extremely important application of artificial intelligence. It has the ability to simultaneously identify multiple object categories in an image and their 2D positions in the image. It is widely used in many fields such as autonomous driving, intelligent security, and industrial manufacturing inspection.

[0003] Single-stage image 2D object detection network models based on Convolutional Neural Networks (CNNs) have become the mainstream image 2D object detection method in industry due to their advantages of low computational complexity (low computational power requirements), performance that is basically consistent with two-stage object detection, standardized operators, and ease of deployment on embedded AI chips. However, when performing inference on images, CNN-based single-stage image 2D object detection network models typically generate a large number of object bounding boxes with confidence scores greater than a certain threshold. These bounding boxes are mostly duplicate bounding boxes corresponding to the same object. Non-Maximum Suppression (NMS), as the final processing step of the object detection output, is used to filter out duplicate bounding boxes that may be located in incorrect positions, thereby improving the accuracy of object detection results. The core principle of NMS is that for multiple bounding boxes with an Intersection over Union (IoU) ratio greater than a specified threshold, only the bounding box with the highest confidence score is retained, that is, filtering duplicate bounding boxes based on confidence scores.

[0004] The standard NMS algorithm has the following problems: 1) The confidence level is usually determined by the target features extracted by the CNN, and is not completely related to the accuracy of the target location. A high confidence level does not mean that the target is in the correct location, which leads to incorrect filtering; 2) The performance of the NMS algorithm is greatly affected by the IoU threshold. An excessively high IoU threshold will cause some duplicate predicted bounding boxes to be unable to be filtered, causing a real target to expand into multiple virtual targets, increasing the false detection rate. On the other hand, an excessively low IoU threshold will easily lead to incorrect filtering of the predicted bounding boxes of adjacent targets in dense target scenes, causing multiple real targets to aggregate into a single virtual target, reducing the detection probability.

[0005] As an important type of sensor in autonomous driving, LiDAR has the ability to measure three-dimensional distance and scan point clouds. It can obtain the distance, orientation and height of the target relative to the LiDAR. It is usually used for the three-dimensional detection of the target - that is, to obtain the target's category and cuboid bounding box. The cuboid bounding box contains the target's distance relative to the LiDAR, the target's orientation, and the target's three-dimensional dimensions of length, width and height.

[0006] Because laser point clouds are sparse and lack necessary texture information, and the parameters of 3D CNNs are sparsely distributed, the computational load and storage space requirements are large. Therefore, the field of autonomous driving typically employs multimodal fusion methods of laser point clouds and 2D images to improve target detection performance. The fusion of laser point clouds and 2D images is generally divided into fusion at the original data level (also known as pre-fusion or data-level fusion), fusion at the CNN feature level (also known as deep fusion or feature-level fusion), and fusion of target detection results (also known as post-fusion or target-level fusion). Furthermore, with in-depth research, fusion methods at different modalities and levels have emerged, such as the fusion of original laser point cloud data and CNN features of 2D images, generally referred to as asymmetric fusion.

[0007] However, current multimodal fusion methods for laser point clouds and 2D images all rely on 3D annotation, i.e., the annotation of targets in the scene as cuboid bounding boxes. This is more time-consuming than traditional 2D rectangular bounding box annotation, significantly increasing the workload of manual annotation and making it difficult to expand the training set in a low-cost and rapid manner, and to increase the generalization ability of target detection (i.e., quickly identify target categories that are not supported in previous datasets). At the same time, it is also impossible to improve target detection performance by utilizing the existing rich open training sets of 2D images.

[0008] For applications such as border patrol and perimeter intrusion target detection, there are significant differences in target density compared to typical autonomous driving scenarios. Targets are typically sparsely distributed in space and time, with fewer targets per frame and less occlusion or clustering between targets. Directly adopting laser point cloud and 2D image multimodal fusion methods from autonomous driving research will lead to numerous problems, including high algorithm complexity, high computational requirements, and low dataset expansion capabilities. Summary of the Invention

[0009] To address at least one technical problem in the prior art, embodiments of the present invention provide a method, apparatus, and storage medium for target detection based on non-maximum suppression-based asymmetric fusion of laser point cloud and image. This method dynamically adjusts the confidence level and IoU filtering threshold of the predicted bounding box in a single-stage image 2D target detection network model, improving the accuracy of 2D target detection, preventing a single real target from expanding into multiple virtual targets, thereby reducing the false detection rate, and also preventing multiple real targets from aggregating into a single virtual target. To achieve the above technical objectives, the technical solution adopted by embodiments of the present invention is as follows:

[0010] In a first aspect, embodiments of the present invention provide a target detection method based on asymmetric fusion of laser point cloud and image using non-maximum suppression, comprising the following steps:

[0011] Step S10: Preset three IoU thresholds: a smaller IoU threshold, a standard IoU threshold, and a larger IoU threshold, denoted as T. sT n With T l ;T s <T n <T l ;

[0012] Step S20: Acquire an image of the target to be detected through a camera, adopt a single-stage image two-dimensional target detection network model, output the target predicted bounding box, including the target category, the target confidence score and the position of the predicted bounding box in the image, filter the predicted bounding boxes below the NMS specified confidence threshold, and classify them into different sets according to the target category;

[0013] Step S30: Acquire laser point cloud using lidar, and project the laser point cloud within the field of view of the relevant camera in the captured image from three-dimensional space to projection space to form a two-dimensional depth map; the pixel value of each pixel in the two-dimensional depth map is depth information, representing the distance between the pixel and the lidar;

[0014] Step S40: Select any set of target categories, and for all the remaining predicted bounding boxes B, count the number N of pixels without depth information in the depth map region corresponding to the predicted bounding box B, and calculate the depth variance V and the depth mean D of the pixels with effective depth information in the depth map region corresponding to the predicted bounding box B.

[0015] If the depth variance V is less than the specified threshold V t And the number of pixels N is less than the specified threshold N t This increases the confidence level of the predicted bounding box B;

[0016] Otherwise, reduce the confidence level of the predicted bounding box B;

[0017] Step S50: After filtering the predicted bounding boxes below the NMS specified confidence threshold again, sort them according to confidence, take out the predicted bounding box B0 with the highest confidence, calculate the IoU between the predicted bounding box B0 and other predicted bounding boxes B1; at the same time, calculate the difference between the average depth D0 in the corresponding depth map region within B0 and the average depth D1 in the corresponding depth map region within B1.

[0018] If the absolute value of the difference is greater than the distance threshold Then choose a larger IoU threshold T l The predicted IoU between bounding boxes B0 and B1 is greater than T. l Only when the predicted bounding box has a lower confidence level should it be filtered out;

[0019] If the absolute value of the difference is less than another distance threshold Then choose a smaller IoU threshold T s The predicted IoU between bounding boxes B0 and B1 is greater than T. sFilter out predicted bounding boxes with lower confidence levels.

[0020] If the absolute value of the difference is and Between these values, the standard IoU threshold T is selected. n The predicted IoU between bounding boxes B0 and B1 is greater than T. n Filter out predicted bounding boxes with lower confidence levels.

[0021] Step S60: Repeat step S50 until there are no predicted bounding boxes in the set of the target category.

[0022] In step S70, for the remaining target categories, repeat steps S40 to S60, and finally output the target detection results.

[0023] Specifically, T s T n With T l Set them to 0.4, 0.5 and 0.6 respectively.

[0024] Specifically, NMS specifies the confidence threshold as 0.1.

[0025] Specifically, in step S40,

[0026] Multiply the confidence score of the predicted bounding box B by To increase confidence;

[0027] Multiply the confidence score of the predicted bounding box B by To reduce confidence level;

[0028] In the formula, C is a set threshold constant.

[0029] Secondly, embodiments of the present invention provide a target detection device based on non-maximum suppression-based asymmetric fusion of laser point cloud and image, comprising:

[0030] Memory, which stores computer programs;

[0031] A processor for running the computer program, which, when running, performs the steps described above.

[0032] Thirdly, embodiments of the present invention provide a storage medium;

[0033] The storage medium stores a computer program configured to execute the steps described above when run.

[0034] Compared with the prior art, the beneficial effects of the technical solution provided by the embodiments of the present invention are as follows:

[0035] 1) To address the sparse target characteristics in application scenarios such as border patrol and perimeter intrusion target detection, an asymmetric fusion method is proposed between the original laser point cloud data and the predicted bounding boxes output by the single-stage image 2D target detection network model without NMS processing. This method eliminates the need to retrain the image 2D target detection network model. By projecting the 3D space to the 2D depth map with low complexity, the relative distance of the target and the density of the target point cloud are estimated. Then, the confidence of the predicted bounding box and the IoU filtering threshold are dynamically adjusted to improve the detection probability of small targets and avoid false detections caused by a real target being inflated into multiple virtual targets.

[0036] 2) Compared with the multimodal fusion method of laser point cloud and 2D image based on the annotation information of 3D cuboid bounding box in the field of autonomous driving, the present invention directly uses the original laser point cloud data, without the need to use the computationally intensive 3D CNN to extract point cloud features, which significantly reduces the computing power requirements of the entire network model; more importantly, it can quickly expand the training dataset through fast and low-cost 2D rectangular annotation, and quickly realize the recognition of target categories that were not previously supported by the image 2D target detection network model. Attached Figure Description

[0037] Figure 1 This is a flowchart of the detection method in an embodiment of the present invention.

[0038] Figure 2 This is a schematic diagram of the detection device in an embodiment of the present invention. Detailed Implementation

[0039] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0040] In the description of the embodiments of the present invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the present invention. In addition, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0041] In the description of the embodiments of the present invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can also refer to the internal connection of two components; and they can refer to a wireless connection or a wired connection. Those skilled in the art can understand the specific meaning of the above terms in the present invention based on the specific circumstances.

[0042] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0043] This application proposes a method for target-sparse scenes, based on a single-stage image 2D target detection network model. It dynamically adjusts the confidence level and IoU filtering threshold of the predicted bounding boxes output by the single-stage image 2D target detection network model according to the original laser point cloud data of the LiDAR, performs non-maximum suppression, and then extracts the real target detection boxes from the multiple output predicted bounding boxes, filtering out non-real target detection boxes. Essentially, it is an asymmetric fusion method of the original laser point cloud data and the prediction results before the image 2D target detection NMS process.

[0044] like Figure 1 As shown in the figure, the target detection method based on non-maximum suppression and asymmetric fusion of laser point cloud and image proposed in this embodiment of the invention includes the following steps:

[0045] Step S10: Preset three IoU thresholds: a smaller IoU threshold, a standard IoU threshold, and a larger IoU threshold, denoted as T. s T n With T l ;T s <T n <T l ;

[0046] In this embodiment, T s T n With T l Set them to 0.4, 0.5, and 0.6 respectively;

[0047] Step S20: Acquire an image of the target to be detected through a camera, adopt a single-stage image two-dimensional target detection network model, output the target predicted bounding box, including the target category, the target confidence score and the position of the predicted bounding box in the image, filter the predicted bounding boxes below the NMS specified confidence threshold, and classify them into different sets according to the target category;

[0048] In this embodiment, the single-stage image two-dimensional object detection network model can adopt the YOLO (You Only Look Once) series model; NMS specifies the confidence threshold as 0.1, and predicted bounding boxes with a confidence level lower than 0.1 are filtered out;

[0049] Step S30: Acquire laser point cloud using lidar, and project the laser point cloud within the field of view of the relevant camera in the captured image from three-dimensional space to projection space to form a two-dimensional depth map; the pixel value of each pixel in the two-dimensional depth map is depth information, representing the distance between the pixel and the lidar;

[0050] In this embodiment, based on the calibration parameters such as the extrinsic and intrinsic parameters of the lidar and camera, and the pinhole imaging model, the lidar point cloud located outside the camera's field of view is filtered out, while the lidar point cloud within the camera's field of view is retained; if a pixel in the two-dimensional depth map does not have depth information, the pixel value is marked as Unknown.

[0051] Step S40, similar to the standard NMS method, arbitrarily select a set of target categories, for all the remaining predicted bounding boxes B, count the number N of pixels without depth information in the depth map region corresponding to the predicted bounding box B, and calculate the depth variance V and the depth mean D of the pixels with effective depth information in the depth map region corresponding to the predicted bounding box B.

[0052] If the depth variance V is less than the specified threshold V t And the number of pixels N is less than the specified threshold N t If the predicted bounding box B is more likely to be a valid target, then increase the confidence level of predicted bounding box B; for example, multiply the confidence level of predicted bounding box B by... To increase confidence, C is a set threshold constant; at this time, both N and V are relatively small, therefore Larger;

[0053] Otherwise, reduce the confidence level of the predicted bounding box B; for example, multiply the confidence level of the predicted bounding box B by... To lower the confidence level; at this point, both N and V are relatively large, therefore Smaller;

[0054] Step S50: After filtering the predicted bounding boxes below the NMS specified confidence threshold again, sort them according to confidence, take out the predicted bounding box B0 with the highest confidence, calculate the IoU between the predicted bounding box B0 and other predicted bounding boxes B1; at the same time, calculate the difference between the average depth D0 in the corresponding depth map region within B0 and the average depth D1 in the corresponding depth map region within B1.

[0055] If the absolute value of the difference is greater than the distance threshold This indicates that the two predicted bounding boxes are far apart, and are more likely to correspond to two different targets. Therefore, a larger IoU threshold T is selected. l The predicted IoU between bounding boxes B0 and B1 is greater than T. l Predicted bounding boxes with lower confidence are filtered out only when necessary to prevent small targets from being suppressed by large targets with higher confidence, thus avoiding missed detection of small targets.

[0056] The existing algorithm sets a fixed IoU threshold, first extracts the predicted bounding box with the highest confidence, calculates the IoU between the predicted bounding box with the highest confidence and other predicted bounding boxes, and then finds those other predicted bounding boxes with a higher IoU threshold, which are filtered out as predicted bounding boxes with lower confidence than the predicted bounding box with the highest confidence.

[0057] If the absolute value of the difference is less than another distance threshold This indicates that the two bounding boxes are close to each other. Considering that target clustering is less common in sparse target scenarios, the two predicted bounding boxes are more likely to correspond to the same target. Therefore, a smaller IoU threshold T is selected. s The predicted IoU between bounding boxes B0 and B1 is greater than T. s Filter out predicted bounding boxes with lower confidence and filter out duplicate predicted bounding boxes to reduce the false detection rate;

[0058] If the absolute value of the difference is and Between these values, the standard IoU threshold T is selected. n The predicted IoU between bounding boxes B0 and B1 is greater than T. n Filter out predicted bounding boxes with lower confidence levels.

[0059] Step S60: Repeat step S50 until there are no predicted bounding boxes in the set of the target category.

[0060] This step is similar to the steps in the standard NMS method;

[0061] Step S70: For the remaining target categories, repeat steps S40 to S60, and finally output the target detection results;

[0062] This step is similar to the steps in the standard NMS method;

[0063] like Figure 2As shown in the figure, this invention also proposes a target detection device based on non-maximum suppression and asymmetric fusion of laser point cloud and image, comprising: a processor and a memory; the processor and the memory communicate with each other, for example, by being connected and communicating with each other through a communication bus; the memory stores a computer program; the processor is used to run the computer program, and the computer program executes the steps described above when it runs; the processor may be a CPU, or other general-purpose processor, digital signal processor (DSP), application specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or other chips, or combinations of the above types of chips or circuits; the memory may include volatile memory, such as random access memory; the memory may also include non-volatile memory, such as flash memory, hard disk or solid-state drive; the memory may also include combinations of the above types of memory;

[0064] This invention also proposes a storage medium storing a computer program configured to execute the steps described above during runtime. The storage medium may be a magnetic disk, optical disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk drive (HDD), or solid-state drive (SSD), etc.; the storage medium may also include combinations of the above types of memory.

[0065] Finally, it should be noted that the above specific embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the technical solutions of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A target detection method based on asymmetric fusion of laser point cloud and image using nonmaximum suppression, characterized in that, Includes the following steps: Step S10: Preset three IoU thresholds: a smaller IoU threshold, a standard IoU threshold, and a larger IoU threshold, denoted as T. s T n With T l ;T s <T n <T l ; Step S20: Acquire an image of the target to be detected through a camera, adopt a single-stage image two-dimensional target detection network model, output the target predicted bounding box, including the target category, the target confidence score and the position of the predicted bounding box in the image, filter the predicted bounding boxes below the NMS specified confidence threshold, and classify them into different sets according to the target category; Step S30: Acquire laser point cloud using lidar, and project the laser point cloud within the field of view of the relevant camera in the captured image from three-dimensional space to projection space to form a two-dimensional depth map; the pixel value of each pixel in the two-dimensional depth map is depth information, representing the distance between the pixel and the lidar; Step S40: Select any set of target categories, and for all the remaining predicted bounding boxes B, count the number N of pixels without depth information in the depth map region corresponding to the predicted bounding box B, and calculate the depth variance V and the depth mean D of the pixels with effective depth information in the depth map region corresponding to the predicted bounding box B. If the depth variance V is less than the specified threshold V t And the number of pixels N is less than the specified threshold N t This increases the confidence level of the predicted bounding box B; Otherwise, reduce the confidence level of the predicted bounding box B; Step S50: After filtering the predicted bounding boxes below the NMS specified confidence threshold again, sort them according to confidence, take out the predicted bounding box B0 with the highest confidence, calculate the IoU between the predicted bounding box B0 and other predicted bounding boxes B1; at the same time, calculate the difference between the average depth D0 in the corresponding depth map region within B0 and the average depth D1 in the corresponding depth map region within B1. If the absolute value of the difference is greater than the distance threshold Then choose a larger IoU threshold T l The predicted IoU between bounding boxes B0 and B1 is greater than T. l Only when the predicted bounding box has a lower confidence level should it be filtered out; If the absolute value of the difference is less than another distance threshold Then choose a smaller IoU threshold T s The predicted IoU between bounding boxes B0 and B1 is greater than T. s Filter out predicted bounding boxes with lower confidence levels. If the absolute value of the difference is and Between these values, the standard IoU threshold T is selected. n The predicted IoU between bounding boxes B0 and B1 is greater than T. n Filter out predicted bounding boxes with lower confidence levels. Step S60: Repeat step S50 until there are no predicted bounding boxes in the set of the target category. In step S70, for the remaining target categories, repeat steps S40 to S60, and finally output the target detection results.

2. The target detection method based on non-maximum suppression and asymmetric fusion of laser point cloud and image as described in claim 1, characterized in that, T s T n With T l Set them to 0.4, 0.5 and 0.6 respectively.

3. The target detection method based on non-maximum suppression and asymmetric fusion of laser point cloud and image as described in claim 1, characterized in that, NMS specifies the confidence threshold as 0.

1.

4. The target detection method based on non-maximum suppression and asymmetric fusion of laser point cloud and image as described in claim 1, characterized in that, In step S40, Multiply the confidence score of the predicted bounding box B by To increase confidence; Multiply the confidence score of the predicted bounding box B by To reduce confidence level; In the formula, C is a set threshold constant.

5. A target detection device based on nonmaximum suppression-based asymmetric fusion of laser point cloud and image, characterized in that, include: Memory, which stores computer programs; A processor for running the computer program, which, when running, performs the steps of the method as described in any one of claims 1 to 4.

6. A storage medium, characterized in that, The storage medium stores a computer program configured to perform the steps of the method as described in any one of claims 1 to 4 when executed.

Images