Target detection method, device, apparatus and storage medium

Abstract

The present disclosure relates to a target detection method, device, equipment and storage medium. The present disclosure obtains the environment point cloud at the current moment and the environment image at the current moment, determines the current contour and the current classification of the candidate target in the environment point cloud at the current moment, obtains the point set associated with the candidate target in the current contour, and obtains the voxelization feature of the point set. Since the environment image has more abundant semantic information, it can more easily distinguish between foreground and background, thereby reducing the problem of target missed detection. By fusing the voxelization feature of the point set and the image feature of the environment image at the current moment, a fused feature is obtained, so that the current contour and the current classification of the candidate target can be corrected according to the fused feature to obtain a correction result, thereby outputting a more accurate target detection result. Prevents unreasonable deceleration, sudden braking and other problems of the movable platform, and can effectively avoid traffic accident hazards.

Description

Technical Field

[0001] This disclosure relates to the field of information technology, and in particular to a target detection method, apparatus, device, and storage medium. Background Technology

[0002] Environmental perception is a crucial component of the algorithm systems used in mobile platforms such as autonomous vehicles, robots, and drones. The core function of environmental perception is object detection. Through object detection, mobile platforms can perceive surrounding objects, which may include pedestrians, other vehicles, cyclists, obstacles, etc. Therefore, the accuracy of object detection is extremely important for environmental perception.

[0003] Typically, mobile platforms can be equipped with lidar, which features high ranging accuracy, rich 3D information, and all-weather capability. Therefore, target detection algorithms based on lidar are widely used.

[0004] However, the inventors of this application have discovered that using lidar for target detection can easily lead to problems such as missed detection, false detection, and inaccurate regression, which can cause problems such as unreasonable deceleration and sudden braking of the mobile platform, and may even pose a traffic accident hazard. Summary of the Invention

[0005] To address, or at least partially address, the aforementioned technical problems, this disclosure provides a target detection method, apparatus, device, and storage medium to output more accurate target detection results. This prevents unreasonable deceleration and sudden braking of mobile platforms and effectively avoids potential traffic accident hazards. Ultimately, it ensures that the mobile platform possesses reliable and stable sensing output capabilities.

[0006] In a first aspect, embodiments of this disclosure provide a target detection method, including:

[0007] Obtain the current environmental point cloud and environmental image;

[0008] Determine the current contour and current classification of the candidate target in the environmental point cloud at the current moment;

[0009] Obtain the set of points associated with the candidate target in the current contour, and obtain the voxelized features of the set of points; fuse the voxelized features of the set of points with the image features of the current environmental image to obtain the fused features;

[0010] Based on the fused features, the current contour and / or current classification of the candidate target are corrected to obtain the corrected result.

[0011] Secondly, embodiments of this disclosure provide a target detection device, comprising:

[0012] The first acquisition module is used to acquire the environmental point cloud and environmental image at the current moment;

[0013] A determination module is used to determine the current contour and current classification of candidate targets in the environmental point cloud at the current moment;

[0014] The second acquisition module is used to acquire a set of points associated with the candidate target in the current contour and obtain the voxelized features of the set of points.

[0015] The fusion module is used to fuse the voxelized features of the point set with the image features of the current environmental image to obtain the fused features;

[0016] The correction module is used to correct the current contour and / or current classification of the candidate target based on the fused features, so as to obtain the correction result.

[0017] Thirdly, embodiments of this disclosure provide an electronic device, including:

[0018] Memory;

[0019] Processor; and

[0020] Computer programs;

[0021] The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first aspect.

[0022] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described in the first aspect.

[0023] The target detection method, apparatus, device, and storage medium provided in this disclosure acquire an environmental point cloud and an environmental image at the current moment. The current contour and current classification of a candidate target are determined from the environmental point cloud. A set of points associated with the candidate target is obtained from the current contour, and the voxelized features of the point set are acquired. Since the environmental image has richer semantic information, it is easier to distinguish between foreground and background, thereby reducing the problem of missed target detection. By fusing the voxelized features of the point set with the image features of the environmental image at the current moment, a fused feature is obtained. Based on the fused feature, at least one of the current contour and current classification of the candidate target is corrected to obtain a corrected result, thus outputting a more accurate target detection result. This prevents unreasonable deceleration and sudden braking of mobile platforms and effectively avoids potential traffic accident hazards. Therefore, it ensures that the mobile platform has reliable and stable perception output capabilities. Attached Figure Description

[0024] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0025] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0026] Figure 1 A schematic diagram illustrating an application scenario provided by an embodiment of this disclosure;

[0027] Figure 2 This is a flowchart of a target detection method provided in an embodiment of the present disclosure;

[0028] Figure 3 A schematic diagram of a point cloud provided for an embodiment of this disclosure;

[0029] Figure 4 This is a flowchart of a target detection method provided in another embodiment of the present disclosure;

[0030] Figure 5 This is a flowchart of a target detection method provided in another embodiment of the present disclosure;

[0031] Figure 6 A schematic diagram of a point cloud provided for another embodiment of this disclosure;

[0032] Figure 7 This is a flowchart of a target detection method provided in another embodiment of the present disclosure;

[0033] Figure 8 This is a schematic diagram of the target detection device provided in an embodiment of the present disclosure;

[0034] Figure 9 A schematic diagram of the structure of an electronic device embodiment provided in this disclosure. Detailed Implementation

[0035] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.

[0036] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.

[0037] Typically, mobile platforms such as autonomous vehicles, robots, and drones are equipped with LiDAR (Light Detection and Ranging) sensors. LiDAR offers advantages such as high ranging accuracy, rich 3D information, and all-weather capability. Therefore, target detection algorithms based on LiDAR are widely adopted. For example... Figure 1 As shown, taking an autonomous vehicle as an example, the lidar on this vehicle can emit detection signals in concentric circles. For example, if each circle of detection signals is considered as a circle, then... Figure 1 As shown by the arrows, the distance between any two adjacent circles increases progressively, and the detection points on the circles become sparser towards the outer edges. In other words, the farther away from the autonomous vehicle, the sparser the detection points become. Therefore, problems such as missed detections, false detections, and inaccurate regression can occur, leading to unreasonable deceleration and sudden braking of the mobile platform, and potentially even posing a traffic accident hazard. To address this problem, this disclosure provides a target detection method, which will be described below with reference to specific embodiments.

[0038] Figure 2 This is a flowchart of a target detection method provided in an embodiment of this disclosure. This method can be applied to environmental perception of mobile platforms such as autonomous vehicles, robots, and drones. Taking an autonomous vehicle as an example, the vehicle can be equipped with a LiDAR and an imaging device, such as a camera or similar device. Specifically, this method can be executed by an in-vehicle device, which can be a car infotainment system, mobile phone, tablet computer, laptop computer, wearable device, etc. This in-vehicle device can be connected to both the LiDAR and the imaging device, enabling it to detect targets based on the point cloud obtained by the LiDAR and the image information obtained by the imaging device. Alternatively, in some embodiments, this method can also be executed by a remote server. For example, the communication module on the autonomous vehicle can send the point cloud obtained by the LiDAR and the image information obtained by the imaging device to a remote server, allowing the remote server to detect targets based on the point cloud and image information. The following example uses an in-vehicle device... Figure 2 As shown, the specific steps of this method are as follows:

[0039] S201. Obtain the environmental point cloud and environmental image at the current moment.

[0040] For example, the mobile platform is Figure 1The autonomous vehicle shown is equipped with a LiDAR and a camera. The LiDAR acquires point clouds of the environment surrounding the autonomous vehicle. The camera captures images of the surrounding environment to obtain image information. For example, the LiDAR can acquire point clouds in real time, and the camera can acquire image information in real time. The latest point cloud acquired by the LiDAR can be recorded as the environmental point cloud at the current moment, and the latest image information acquired by the camera can be recorded as the environmental image at the current moment. It is understood that the LiDAR can periodically acquire point clouds at a first preset time interval, and the camera can periodically acquire image information at a second preset time interval. The first and second preset time intervals can be the same or different. Furthermore, the onboard equipment in the autonomous vehicle can acquire the environmental point cloud at the current moment from the LiDAR and the environmental image at the current moment from the camera.

[0041] S202. Determine the current contour and current classification of the candidate target in the environmental point cloud at the current moment.

[0042] For example, Figure 3 The diagram shown is a schematic of the environmental point cloud at the current moment. It can be understood that the environmental point cloud obtained by the lidar at different times may be different. Figure 3 The coordinate system of the current environmental point cloud shown can be a coordinate system with the center of the LiDAR as the origin. This coordinate system can be a three-dimensional coordinate system. Each point in the current environmental point cloud corresponds to a three-dimensional coordinate, and each point in the current environmental point cloud can correspond to a sampling point in the surrounding environment detected by the LiDAR. Furthermore, the three-dimensional coordinates of each point in the current environmental point cloud can represent the spatial coordinates of a sampling point, and the distance between this three-dimensional coordinate and the origin is the distance of that sampling point from the LiDAR. In other words, after obtaining the spatial coordinates of each sampling point in the environment surrounding the autonomous vehicle, a set of points is obtained, which is called a point cloud.

[0043] For example, in-vehicle equipment can perform point cloud clustering on the current environmental point cloud or a subset of points within it, obtaining information such as the current length, width, height, center point, and angle of candidate targets in the current environmental point cloud. The number of candidate targets can be one or more. The angle can be the angle of the candidate target relative to the LiDAR. Furthermore, based on the current length, width, height, center point, and angle information of the candidate target, the in-vehicle equipment can obtain the current 3D bounding box, the current classification, and the points within the current 3D bounding box of the candidate target. The current 3D bounding box of the candidate target can be the current outline of the candidate target. For example... Figure 3The diagram shows the current contour of candidate target 1 (box 30), candidate target 2 (box 31), and candidate target 3 (box 32). Candidate target 1 is currently classified as an obstacle, candidate target 2 as a road edge, and candidate target 3 as a road edge. Furthermore, points within box 30 can be considered sampling points on the obstacle surface, and points within boxes 31 and 32 can be considered sampling points on the road edge surface.

[0044] Optionally, the candidate targets are determined in the following ways: based on target points in the environmental point cloud at the current time with a confidence level less than a first threshold; or based on target points in the environmental point cloud at the current time with a confidence level greater than or equal to the first threshold, and some foreground points; or based on the three-dimensional detection results of the environmental image at the current time.

[0045] For example Figure 4 The diagram shows the algorithm system framework of the in-vehicle device. This framework includes three modules: a candidate target generation module 41, an abnormal target filtering and correction module 42, and a target fusion module. The candidate target generation module 41 generates candidate targets. For example, the detection model in the candidate target generation module 41 can output the probability value that each point in the current environmental point cloud is a target point. If the probability value of a point in the current environmental point cloud being a target point is greater than a threshold, it means that the point in the current environmental point cloud corresponds to a sampling point on the surface of a target object in the environment surrounding the autonomous vehicle. This target object can be a pedestrian, other vehicles, cyclists, obstacles, etc. Therefore, based on the probability value that each point in the current environmental point cloud is a target point, the points in the current environmental point cloud can be divided into high-confidence target points and low-confidence target points. Among them, high-confidence target points can be points in the environmental point cloud at the current time whose probability value is greater than the first threshold, and low-confidence target points can be points in the environmental point cloud at the current time whose probability value is less than the second threshold. The first threshold and the second threshold can be the same or different. If they are different, the first threshold can be greater than the second threshold.

[0046] In the first feasible implementation, low-confidence target points are extracted from the environmental point cloud at the current moment. Then, based on the prior knowledge that target objects generally exist on the road surface, the low-confidence target points are filtered by retrieving drivable areas (ROIs) containing roads and intersections through a high-precision map, thereby retaining the points on the road surface among the low-confidence target points. Furthermore, point cloud clustering is performed on the points on the road surface among the low-confidence target points to generate candidate targets.

[0047] In the second feasible implementation, point cloud clustering is performed on points in the current environmental point cloud that do not belong to low probability values ​​and some foreground points to generate candidate targets.

[0048] In a third feasible implementation, the 3D detection result of the current environmental image is used as a candidate target. This 3D detection result can be a 3D bounding box that includes the target point. This bounding box can be in the same coordinate system as the point cloud.

[0049] The candidate targets obtained through the three methods described above can be different; if they are the same, deduplication can be performed. It is understood that the three methods of generating candidate targets can be executed in parallel, allowing the on-board device to detect multiple candidate targets simultaneously. This enables the candidate target generation module 41 to output the current contour and current classification of each candidate target. In other embodiments, the candidate target generation module 41 can output the current contour, current classification, and points within the current contour of each candidate target. Each candidate target can correspond to a channel, and the candidate target generation module 41 can provide the current contour, current classification, and points within the current contour of a candidate target to the abnormal target filtering and correction module 42 through one channel. In other words, the candidate target generation module 41 can provide the relevant information of multiple candidate targets (e.g., current contour, current classification, and points within the current contour) to the abnormal target filtering and correction module 42 in parallel through multiple channels. This allows the abnormal target filtering and correction module 42 to process the relevant information of multiple candidate targets in parallel. Alternatively, in some other embodiments, the candidate target generation module 41 can provide the relevant information of multiple candidate targets serially to the abnormal target filtering and correction module 42. This allows the abnormal target filtering and correction module 42 to process the relevant information of multiple candidate targets serially.

[0050] The following example illustrates how the abnormal target screening and correction module 42 processes information related to a candidate target.

[0051] S203. Obtain the set of points associated with the candidate target in the current contour, and obtain the voxelized features of the set of points.

[0052] For example Figure 4 The specific structure of the abnormal target filtering and correction module 42 shown can be as follows: Figure 5 The structure shown. Among them, Figure 4 The target deterministic classification module, target reclassification module, and target reregression module correspond to... Figure 5 Instance branches in Figure 4 Voxelization regression corresponds to Figure 5 Point branches in the middle.

[0053] The core of the anomaly target screening and correction module 42 is FADNet, a second-order anomaly detection neural network model based on multimodal fusion. The input to FADNet is as follows: Figure 5 The diagram shows the point set and image features. The point set is the set of points associated with the candidate target, obtained from the environmental point cloud at the current time based on the candidate target's current contour. The image features can be the image features of the environmental image at the current time, as described above. Figure 5 As shown, the point set is voxelized to obtain its voxelized features.

[0054] S204. The voxelized features of the point set and the image features of the current environmental image are fused to obtain the fused features.

[0055] like Figure 5 As shown, the result of performing a two-dimensional convolution on the image features can also be an image feature, but the image features after two-dimensional convolution have richer information. Furthermore, the voxelized features of the point set and the image features after two-dimensional convolution are fused to obtain the fused features.

[0056] The point set can be a set of points determined from the environmental point cloud at the current moment. Alternatively, the point set can be a set of points determined from a richer point cloud in the coordinate system of the current environmental point cloud after mapping the historical point cloud to the coordinate system of the current environmental point cloud.

[0057] S205. Based on the fused features, the current contour and / or current classification of the candidate target are corrected to obtain the correction result.

[0058] like Figure 5 As shown, the fused features undergo further feature extraction via a 3D sparse convolutional encoder and decoder. The output of the 3D sparse convolutional decoder is still the fused features, but with richer information. Furthermore, the instance branch can determine the current contour and classification of the candidate target based on the fused features output by the 3D sparse convolutional decoder, such as... Figure 5As shown, the input to the instance branch includes the fused features output by the 3D sparse convolutional decoder, and the current contour and current classification of the candidate target output by the candidate target generation module 41. This instance branch includes a classifier that can output one of three results: normal, false positive, or inaccurate regression. Specifically, when both the current contour and current classification of the candidate target are correct, the classifier outputs "normal." When the current classification of the candidate target is incorrect, the classifier outputs "false positive." When the current contour of the candidate target is incorrect, the classifier outputs "inaccurate regression." In other words, the output of the classifier can be the judgment result output by the instance branch.

[0059] like Figure 5 As shown, the point branch can determine the points within the current contour based on the fused features output by the 3D sparse convolutional decoder. For example, the input of the point branch includes the fused features output by the 3D sparse convolutional decoder and the points within the current contour of the candidate target output by the candidate target generation module 41. By default, the probability value of each point within the current contour output by the candidate target generation module 41 being a target point is greater than a first threshold, that is, by default, the points within the current contour output by the candidate target generation module 41 correspond to the sampling points on the surface of the target object in the environment surrounding the autonomous vehicle. Specifically, the point branch can determine whether each point within the current contour output by the candidate target generation module 41 corresponds to the sampling points on the surface of the target object in the environment surrounding the autonomous vehicle. If more than 50% of the points within the current contour do not correspond to the sampling points on the surface of the target object in the environment surrounding the autonomous vehicle, the point branch can output an incorrect judgment result. If more than 90% of the points within the current contour correspond to the sampling points on the surface of the target object in the environment surrounding the autonomous vehicle, the point branch can output a correct judgment result and output which points within the current contour do not correspond to the sampling points on the surface of the target object in the environment surrounding the autonomous vehicle. It is understood that the 50% and 90% mentioned here are only illustrative of thresholds. In other embodiments, the type of threshold and the specific value of the threshold are not limited.

[0060] like Figure 5 The result fusion module shown can fuse the judgment results output by the instance branch and the judgment results output by the point branch. For example, it can modify at least one of the following: the current contour, the current classification, and the points within the current contour of the candidate target, based on the judgment results output by the instance branch and the point branch, to obtain a modified result.

[0061] This embodiment acquires the current environmental point cloud and the current environmental image. The current contour and current classification of a candidate target are determined from the current environmental point cloud. A set of points associated with the candidate target is obtained from the current contour, and the voxelized features of the point set are acquired. Since the environmental image has richer semantic information, it is easier to distinguish between foreground and background, thereby reducing the problem of missed target detection. By fusing the voxelized features of the point set with the image features of the current environmental image, a fused feature is obtained. Based on the fused feature, at least one of the current contour and current classification of the candidate target can be corrected to obtain a corrected result, thus outputting a more accurate target detection result. This prevents unreasonable deceleration and sudden braking of the mobile platform and effectively avoids potential traffic accident hazards. Therefore, it ensures that the mobile platform has reliable and stable perception output capabilities.

[0062] In one possible implementation, obtaining the set of points associated with the candidate target in the current contour includes: obtaining points within the current contour and surrounding environmental points of the candidate target in the environmental point cloud at the current moment, and using the points within the current contour and surrounding environmental points of the candidate target as the set of points associated with the candidate target.

[0063] like Figure 6 As shown, box 30 represents the current contour of candidate target 1. Based on the current contour, a set of points associated with candidate target 1 can be obtained from the environmental point cloud at the current moment. This set of points includes not only the points within box 30 but also the surrounding environmental points of candidate target 1. These surrounding environmental points can be points outside box 30 that are within a preset range from box 30. For example, this set of points is... Figure 6 The set shown is 60.

[0064] In another possible implementation, before obtaining the point set associated with the candidate target in the current contour, the method further includes: mapping the environmental point cloud at a historical moment to the coordinate system where the environmental point cloud at the current moment is located, to obtain a fused point cloud; correspondingly, obtaining the point set associated with the candidate target in the current contour includes: obtaining the points in the current contour and the surrounding environmental points of the candidate target in the fused point cloud, and using the points in the current contour and the surrounding environmental points of the candidate target as the point set associated with the candidate target.

[0065] For example, the current time is denoted as t. t-1, t-2, ..., tn are denoted as historical times, respectively. The point cloud obtained by the lidar at the current time is denoted as the environmental point cloud at the current time. The point clouds obtained by the lidar at historical times are denoted as historical point clouds. For example, the point cloud obtained by the lidar at historical time t-1 is denoted as the first historical point cloud, the point cloud obtained by the lidar at historical time t-2 is denoted as the second historical point cloud, and so on, with the point cloud obtained by the lidar at historical time tn being denoted as the nth historical point cloud. In this embodiment, the first historical point cloud, the second historical point cloud, ..., the nth historical point cloud can also be mapped sequentially to the coordinate system of the environmental point cloud at the current time, thereby making the coordinate system of the environmental point cloud at the current time contain a richer point cloud. Here, the point cloud mapped from the historical point cloud to this coordinate system and the original environmental point cloud at the current time in this coordinate system can be denoted as the fused point cloud. Furthermore, based on the current contour of the candidate target, a set of points associated with the candidate target is obtained from the fused point cloud. Similarly, this set of points can include points within the current contour and points in the surrounding environment of the candidate target. However, the number of points contained in this set should be, for example... Figure 6 The set 60 shown contains a richer variety of points.

[0066] This embodiment obtains a fused point cloud by mapping historical point clouds to the coordinate system of the current environmental point cloud. Based on the current contour, a set of points associated with the candidate target is obtained from the fused point cloud, thus enriching the point set. Since the detection points become sparser the farther away from the autonomous vehicle, target objects far from the autonomous vehicle may be detected at some times and not at others. By mapping historical point clouds to the coordinate system of the current environmental point cloud, point clouds from different times can be superimposed, thereby further and more effectively preventing missed detections.

[0067] Based on the above embodiments, the step of correcting the current contour and / or current classification of the candidate target according to the fused features to obtain a correction result includes, for example: Figure 7 The following steps are shown:

[0068] S701. Based on the fused features, redetermine the contour and classification of the candidate targets.

[0069] like Figure 5 As shown, the 3D sparse convolutional decoder in FADNet can output fused features. Furthermore, as... Figure 4As shown, the target reclassification module can redetermine the classification of candidate targets. The target regression module can redetermine the contour of the candidate targets and calculate the offset between the redetermined contour and the current contour of the candidate targets output by the candidate target generation module 41. In some embodiments, the contour of the candidate targets redetermined by the target regression module may be the actual contour of the candidate targets.

[0070] S702. If the offset between the redefined contour and the current contour is greater than the second threshold, the current contour is corrected according to the offset to obtain the corrected contour.

[0071] Specifically, such as Figure 4 The target deterministic classification module shown can specifically be a classifier as described above. This classifier can output one of three results: normal, false positive, or inaccurate regression. For example, if the offset between the contour of the candidate target re-determined by the target re-regression module and the current contour of the candidate target output by the candidate target generation module 41 is greater than the second threshold, it indicates that the current contour of the candidate target is inaccurate. In this case, the classifier outputs "inaccurate regression." Figure 5 The result fusion module shown can obtain the offset from the target re-regression module based on the regression inaccuracy output by the classifier, and correct the current contour based on the offset to obtain the corrected contour.

[0072] S703. If the redefined classification is inconsistent with the current classification, the current classification is corrected to obtain the corrected classification.

[0073] For example, when the classification of a candidate target redefined by the target reclassification module is inconsistent with the current classification of the candidate target output by the candidate target generation module 41, the classifier outputs a false detection. At this time, the result fusion module can obtain the redefined classification of the candidate target from the target reclassification module based on the false detection output by the classifier, and use the redefined classification as the corrected classification for the current classification.

[0074] Optionally, if the redefined classification is inconsistent with the current classification, the candidate target can be deleted or labeled.

[0075] For example, when the classifier outputs a false detection, it means that the redefined classification is inconsistent with the current classification. In this case, the candidate target can be deleted, or the candidate target can be labeled, for example, as an uncertain target.

[0076] When the classifier outputs an inaccurate regression, the candidate target may not be deleted, but its current contour may be corrected based on the offset described above.

[0077] S704. Based on the fused features, redetermine whether each point within the corrected contour or the current contour is a target point.

[0078] For example Figure 4 The voxelization re-regression module shown can re-evaluate each point within the current contour of the candidate target output by the candidate target generation module 41. For example, it can determine whether each point is a target point. If a point is a target point, it means that the point corresponds to a sampling point on the surface of the target object in the environment surrounding the autonomous vehicle. If a point is not a target point, it means that the point does not correspond to a sampling point on the surface of the target object in the environment surrounding the autonomous vehicle.

[0079] Alternatively, if the current contour is corrected, the voxelization re-regression module can re-evaluate each point within the corrected contour to determine whether each point within the corrected contour is the target point.

[0080] The correction result includes at least one of the following: the corrected contour, the corrected classification, the corrected contour, or points within the current contour after excluding non-target points. For example, if Figure 5 The example branch output is normal, and the dot branch output is correct, then as follows: Figure 5 The fusion results generated by the fusion module shown include: the current contour, current classification, and points within the current contour of the candidate target output by the candidate target generation module 41.

[0081] if Figure 5 If the example branch outputs a false positive, and the point branch outputs an error, then the candidate target is deleted or labeled.

[0082] if Figure 5 If the example branch output regression is inaccurate and the point branch output is incorrect, the fusion result generated by the result fusion module includes: the current classification of the candidate target output by the candidate target generation module 41, the corrected contour, and the points in the corrected contour or the current contour after excluding non-target points.

[0083] if Figure 5 If the example branch output is normal and the point branch output is incorrect, the fusion result generated by the result fusion module includes: the current contour, current classification, and points within the current contour after excluding non-target points, as output by the candidate target generation module 41.

[0084] if Figure 5 The example branch outputs false positives, while the point branch outputs correct positives. The fusion result generated by the result fusion module includes: the corrected classification, the current contour of the candidate target, and the points within the current contour.

[0085] if Figure 5The example branch output shows inaccurate regression, while the point branch output is correct. In this case, the current contour of the candidate target can be marked to indicate that the candidate target is an uncertain target. Alternatively, the candidate target can be deleted. Alternatively, the result fusion module can output the current classification of the candidate target, the corrected contour, and the points within the corrected contour after excluding non-target points.

[0086] In addition, the method further includes: if the correction result matches the reference information successfully, then output the correction result; if the correction result fails to match the reference information, then annotate the correction result or do not output the correction result; wherein, the reference information includes at least one of the three-dimensional detection results of the current environmental image, the historical contour of the candidate target, and the detection results of the millimeter-wave radar.

[0087] like Figure 4 As shown, the input to the target fusion module includes at least two of the following: the fusion result generated by the result fusion module, the high-confidence target point, and the 3D detection result of the current environmental image. The fusion result generated by the result fusion module can be referred to as the correction result.

[0088] When the input to the target fusion module includes the fusion result generated by the result fusion module and the high-confidence target point, the correlation between the fusion result generated by the result fusion module and the high-confidence target point is not strong. The target fusion module can directly output the fusion result generated by the result fusion module and the high-confidence target point.

[0089] When the input to the target fusion module includes the fusion result generated by the result fusion module and the 3D detection result of the current environmental image, the target fusion module can fuse these two results due to their strong correlation. For example, the fusion result generated by the result fusion module may include the contours (e.g., 3D bounding boxes) of candidate targets, and the 3D detection result of the current environmental image may also include the contours (e.g., 3D bounding boxes) of candidate targets. If the overlap between these two contours is high, it indicates a match, and the target fusion module can output the fusion result generated by the result fusion module. If the overlap between these two contours is low, the target fusion module can label the candidate target to indicate that it is an uncertain target, or the target fusion module may not output anything.

[0090] If the fusion result generated by the result fusion module includes the outline of the candidate target, but the 3D detection result of the current environmental image does not include the outline of the candidate target, the target fusion module can mark the candidate target to indicate that the candidate target is an uncertain target.

[0091] In some other embodiments, the input to the target fusion module may also include historical profiles of candidate targets and / or detection results from millimeter-wave radar.

[0092] For example, when the fusion result generated by the result fusion module includes the outline of the candidate target, and the outline matches at least one of the three-dimensional detection results of the current environmental image, the historical outline of the candidate target, and the detection results of the millimeter-wave radar, the target fusion module can output the fusion result generated by the result fusion module.

[0093] If the fusion result generated by the result fusion module includes the outline of the candidate target, and the outline does not match any of the three-dimensional detection results of the current environmental image, the historical outline of the candidate target, or the detection results of the millimeter-wave radar, then the target fusion module can mark the candidate target to indicate that the candidate target is an uncertain target.

[0094] This embodiment of the disclosure fuses the voxelized features of the point set associated with the candidate target with the image features of the current environmental image to obtain fused features. The fused features include not only the point cloud components of the candidate target but also its image features. Therefore, based on the fused features, the current contour, current classification, and points within the current contour of the candidate target can be accurately corrected. Furthermore, the corrected result can be fused with at least one of the following: the 3D detection results of the current environmental image, the historical contour of the candidate target, and the detection results of millimeter-wave radar, thereby outputting a more accurate target detection result. Therefore, compared to existing LiDAR target detection schemes that primarily use deep neural network models, this embodiment of the disclosure does not require optimization of the deep neural network model; for example, it does not require modification of the model structure, addition of training data, or optimization of model training parameters. While optimizing deep neural network models can improve target detection to some extent, this optimization process is difficult to overcome the inherent characteristics of LiDAR (such as target missed detection, false detection, and inaccurate regression). The embodiments disclosed in this paper can not only avoid target missed detection, false detection, and inaccurate regression, but also avoid the problems of long cycle and unclear expected results caused by optimizing deep neural network models.

[0095] Furthermore, the embodiments disclosed herein are not in the first-order point cloud detection stage (e.g.) Figure 4 The detection model shown fuses point clouds and images, and this fusion requires accurate calibration and a smooth road surface. This embodiment of the present disclosure fuses voxelized features of a point set composed of partial points in the point cloud and image features within the second-order anomaly detection neural network model FADNet. Therefore, it does not require strict usage conditions, thus improving its applicability.

[0096] Figure 8 This is a schematic diagram of the target detection device provided in an embodiment of the present disclosure. The target detection device provided in this embodiment of the present disclosure can execute the processing flow provided in the target detection method embodiment, such as... Figure 8 As shown, the target detection device 80 includes:

[0097] The first acquisition module 81 is used to acquire the environmental point cloud and environmental image at the current moment;

[0098] The determination module 82 is used to determine the current contour and current classification of the candidate target in the environmental point cloud at the current moment;

[0099] The second acquisition module 83 is used to acquire a set of points associated with the candidate target in the current contour and obtain the voxelized features of the set of points.

[0100] The fusion module 84 is used to fuse the voxelized features of the point set and the image features of the current environmental image to obtain the fused features;

[0101] The correction module 85 is used to correct the current contour and / or current classification of the candidate target based on the fused features to obtain a correction result.

[0102] Optionally, the determining module 82 is further configured to determine the candidate target in the following manner:

[0103] The target points are determined based on their confidence level being less than a first threshold in the current environmental point cloud; or...

[0104] The target points in the current environmental point cloud with a confidence level greater than or equal to a first threshold, and some foreground points, are determined; or

[0105] Determined based on the 3D detection results of the environmental image at the current moment.

[0106] Optionally, when the second acquisition module 83 acquires the set of points associated with the candidate target in the current contour, it is specifically used for:

[0107] In the current moment, obtain the points within the current contour and the surrounding environment points of the candidate target from the environmental point cloud, and use the points within the current contour and the surrounding environment points of the candidate target as the point set associated with the candidate target.

[0108] Optionally, the target detection device 80 further includes a mapping module 86, which is used to map the environmental point cloud at a historical time to the coordinate system of the environmental point cloud at the current time before the second acquisition module 83 acquires the point set associated with the candidate target in the current contour, thereby obtaining a fused point cloud; when the second acquisition module 83 acquires the point set associated with the candidate target in the current contour, it is specifically used for:

[0109] The points within the current contour and the surrounding environment points of the candidate target are obtained from the fused point cloud, and the points within the current contour and the surrounding environment points of the candidate target are used as the point set associated with the candidate target.

[0110] Optionally, the correction module 85 corrects the current contour and / or current classification of the candidate target based on the fused features, and when obtaining the correction result, it is specifically used for:

[0111] Based on the fused features, the outline and classification of the candidate targets are redefined;

[0112] If the offset between the redefined contour and the current contour is greater than the second threshold, the current contour is corrected according to the offset to obtain the corrected contour.

[0113] If the redefined classification is inconsistent with the current classification, the current classification is corrected to obtain the corrected classification.

[0114] Optionally, the determining module 82 is also used for:

[0115] Based on the fused features, it is re-determined whether each point within the corrected contour or the current contour is a target point;

[0116] The correction result includes at least one of the following: the corrected contour, the corrected classification, the corrected contour, or points within the current contour after excluding non-target points.

[0117] Optionally, the target detection device 80 further includes a deletion module 87, used to delete the candidate target or label the candidate target if the redefined classification is inconsistent with the current classification.

[0118] Optionally, the target detection device 80 further includes: an output module 88 and an annotation module 89; wherein, the output module 88 is used to output the correction result if the correction result matches the reference information successfully; the annotation module 89 is used to annotate the correction result if the correction result fails to match the reference information, or not to output the correction result.

[0119] The reference information includes at least one of the following: the three-dimensional detection result of the current image information, the historical contour of the candidate target, and the detection result of the millimeter-wave radar.

[0120] Figure 8 The target detection device shown in the embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and will not be repeated here.

[0121] The above describes the internal functions and structure of the target detection device, which can be implemented as an electronic device. Figure 9 A schematic diagram illustrating the structure of an electronic device embodiment provided in this disclosure. (See attached diagram.) Figure 9 As shown, the electronic device includes a memory 91 and a processor 92.

[0122] Memory 91 is used to store programs. In addition to the programs described above, memory 91 can also be configured to store various other data to support operation on the electronic device. Examples of this data include instructions for any application or method used to operate on the electronic device, contact data, phonebook data, messages, pictures, videos, etc.

[0123] The memory 91 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.

[0124] The processor 92 is coupled to the memory 91 and executes the program stored in the memory 91 for:

[0125] Obtain the current environmental point cloud and environmental image;

[0126] Determine the current contour and current classification of the candidate target in the environmental point cloud at the current moment;

[0127] Obtain the set of points associated with the candidate target in the current contour, and obtain the voxelized features of the set of points; fuse the voxelized features of the set of points with the image features of the current environmental image to obtain the fused features;

[0128] Based on the fused features, the current contour and / or current classification of the candidate target are corrected to obtain the corrected result.

[0129] Furthermore, such as Figure 9 As shown, the electronic device may also include other components such as a communication component 93, a power supply component 94, an audio component 95, and a display 96. Figure 9 The diagram only shows some components and does not mean that the electronic device includes only these components. Figure 9 The components shown.

[0130] Communication component 93 is configured to facilitate wired or wireless communication between electronic devices and other devices. The electronic devices can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 93 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 93 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0131] Power supply component 94 provides power to various components of the electronic device. Power supply component 94 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device.

[0132] Audio component 95 is configured to output and / or input audio signals. For example, audio component 95 includes a microphone (MIC) configured to receive external audio signals when the electronic device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 91 or transmitted via communication component 93. In some embodiments, audio component 95 also includes a speaker for outputting audio signals.

[0133] Display 96 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.

[0134] In addition, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the target detection method described in the above embodiments.

[0135] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0136] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A target detection method, wherein, The method includes: Obtain the current environmental point cloud and environmental image; Determine the current contour and current classification of the candidate target in the environmental point cloud at the current moment; Obtain the set of points associated with the candidate target in the current contour, and obtain the voxelized features of the set of points; fuse the voxelized features of the set of points with the image features of the current environmental image to obtain the fused features; Based on the fused features, the current contour and / or current classification of the candidate target are corrected to obtain a correction result. This includes: fusing the judgment results output by the instance branch and the judgment results output by the point branch to correct the current contour and / or current classification of the candidate target to obtain a correction result. The instance branch judges the current contour and current classification of the candidate target based on the fused features, and the point branch judges whether the points of the current contour of the candidate target correspond to the sampling points of the object surface in the current environment based on the fused features.

2. The method according to claim 1, wherein, The candidate targets are determined in the following manner: The target points are determined based on their confidence level being less than a first threshold in the current environmental point cloud; or... The target points in the current environmental point cloud with a confidence level greater than or equal to a first threshold, and some foreground points, are determined; or Determined based on the 3D detection results of the environmental image at the current moment.

3. The method according to claim 1 or 2, wherein, The step of obtaining the set of points associated with the candidate target in the current contour includes: In the current moment, obtain the points within the current contour and the surrounding environment points of the candidate target from the environmental point cloud, and use the points within the current contour and the surrounding environment points of the candidate target as the point set associated with the candidate target.

4. The method according to claim 3, wherein, Before obtaining the set of points associated with the candidate target in the current contour, the method further includes: The environmental point cloud at a historical moment is mapped to the coordinate system of the environmental point cloud at the current moment to obtain the fused point cloud; Accordingly, obtaining the set of points associated with the candidate target in the current contour includes: The points within the current contour and the surrounding environment points of the candidate target are obtained from the fused point cloud, and the points within the current contour and the surrounding environment points of the candidate target are used as the point set associated with the candidate target.

5. The method according to claim 1, wherein, The step of correcting the current contour and / or current classification of the candidate target based on the fused features to obtain a correction result includes: Based on the fused features, the outline and classification of the candidate targets are redefined; If the offset between the redefined contour and the current contour is greater than the second threshold, the current contour is corrected according to the offset to obtain the corrected contour. If the redefined classification is inconsistent with the current classification, the current classification is corrected to obtain the corrected classification.

6. The method according to claim 5, wherein, The method further includes: Based on the fused features, it is re-determined whether each point within the corrected contour or the current contour is a target point; The correction result includes at least one of the following: the corrected contour, the corrected classification, the corrected contour, or points within the current contour after excluding non-target points.

7. The method according to claim 5, wherein, The method further includes: If the redefined classification is inconsistent with the current classification, the candidate target is deleted or labeled.

8. The method according to claim 1, wherein, The method further includes: If the correction result matches the reference information successfully, then the correction result is output; If the correction result fails to match the reference information, the correction result is marked, or the correction result is not output. The reference information includes at least one of the following: the three-dimensional detection results of the current environmental image, the historical contours of the candidate target, and the detection results of the millimeter-wave radar.

9. A target detection device, wherein, include: The first acquisition module is used to acquire the environmental point cloud and environmental image at the current moment; A determination module is used to determine the current contour and current classification of candidate targets in the environmental point cloud at the current moment; The second acquisition module is used to acquire a set of points associated with the candidate target in the current contour and obtain the voxelized features of the set of points. The fusion module is used to fuse the voxelized features of the point set with the image features of the current environmental image to obtain the fused features; The correction module is used to correct the current contour and / or current classification of the candidate target based on the fused features to obtain a correction result. The correction of the current contour and / or current classification of the candidate target based on the fused features to obtain a correction result includes: fusing the judgment results output by the instance branch and the judgment results output by the point branch to correct the current contour and / or current classification of the candidate target to obtain a correction result; the instance branch judges the current contour and current classification of the candidate target based on the fused features; and the point branch judges whether the points of the current contour of the candidate target correspond to sampling points on the surface of an object in the current environment based on the fused features.

10. An electronic device, wherein, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-8.

11. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-8.

Images