Target detection method and electronic device

Abstract

The application provides a target detection method and an electronic device, and relates to the technical field of unmanned driving. The method comprises the following steps: performing target detection on target image data to obtain image detection results corresponding to at least one target to be detected; based on the image detection results, obtaining a first point cloud set corresponding to the at least one target to be detected from a point cloud map constructed by a plurality of target point cloud data corresponding to the target image data; performing fusion processing on a plurality of first point cloud sets to obtain a second point cloud set, wherein the plurality of first point cloud sets comprise first point cloud sets obtained based on a plurality of target image data respectively; and determining a target point cloud set corresponding to the at least one target to be detected based on the second point cloud set. For the scene of unmanned driving, automatic driving and unmanned vehicles, the embodiment of the application can realize high-recall-rate and high-precision detection of small targets.

Description

Technical Field

[0001] This application relates to the field of autonomous driving technology, specifically to a target detection method and electronic device. Background Technology

[0002] With the development of autonomous driving technology, many autonomous driving scenarios require the detection of small objects on the road that could endanger safety, such as gravel, ruts, and potholes, in order to avoid these small objects in time during autonomous driving and improve the safety of autonomous driving.

[0003] Traditional target detection schemes typically rely on sensor data from a single modality for target detection. As a result, the detection of small targets may suffer from low recall and low accuracy. Summary of the Invention

[0004] In view of this, the embodiments of this application aim to provide a target detection method to solve the problems of low target recall and low detection accuracy when detecting small targets in the prior art.

[0005] In a first aspect, one embodiment of this application provides a target detection method, comprising: performing target detection on target image data to obtain an image detection result corresponding to at least one target to be detected; based on the image detection result, obtaining a first point cloud set corresponding to the at least one target to be detected from a point cloud map constructed from multiple frames of target point cloud data corresponding to the target image data; performing fusion processing on multiple first point cloud sets to obtain a second point cloud set, wherein the multiple first point cloud sets include first point cloud sets obtained based on multiple target image data respectively; and determining a target point cloud set corresponding to the at least one target to be detected based on the second point cloud set.

[0006] In conjunction with the first aspect, in some implementations of the first aspect, based on the image detection result, obtaining a first point cloud set corresponding to the at least one target from a point cloud map constructed from multiple frames of target point cloud data corresponding to the target image data includes: for each frame of target point cloud data in the multiple frames of target point cloud data corresponding to the target image data, projecting the target point cloud data from the radar coordinate system to the world coordinate system based on initial transformation parameters to obtain multiple frames of target point cloud data in the world coordinate system; performing fusion processing on the multiple frames of target point cloud data in the world coordinate system to obtain the point cloud map; and obtaining the first point cloud set corresponding to the at least one target from the point cloud map based on the image detection result.

[0007] In conjunction with the first aspect, in some implementations of the first aspect, the multi-frame target point cloud data includes current frame target point cloud data and historical frame target point cloud data; the step of fusing the multi-frame target point cloud data in the world coordinate system to obtain the point cloud map includes: acquiring a historical point cloud map and a historical feature point cloud map constructed based on the historical frame target point cloud data in the world coordinate system; extracting features from the current frame target point cloud data to obtain a feature point cloud corresponding to the current frame target point cloud data; performing adjustment processing on the feature point cloud corresponding to the current frame target point cloud data based on the historical feature point cloud map to optimize the initial transformation parameters and obtain target transformation parameters; reprojecting the current frame target point cloud data to the world coordinate system based on the target transformation parameters to obtain reprojected current frame target point cloud data; and fusing the reprojected current frame target point cloud data with the historical point cloud map to obtain the point cloud map.

[0008] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: fusing the feature point cloud corresponding to the reprojected current frame target point cloud data with the historical feature point cloud map to obtain a feature point cloud map.

[0009] In conjunction with the first aspect, in some implementations of the first aspect, the step of fusing the feature point cloud corresponding to the reprojected current frame target point cloud data with the historical feature point cloud map to obtain a feature point cloud map includes: dividing the feature point cloud space corresponding to the historical feature point cloud map into voxels to obtain multiple voxels; performing the following processing on each of the multiple voxels: determining the feature information corresponding to the voxel based on the feature points of the reprojected current frame target point cloud data in the voxel and the feature points of the historical feature point cloud map in the voxel; if the feature information corresponding to the voxel is target feature information, then fusing the feature points in the voxel; if the feature information corresponding to the voxel is not target feature information, then dividing the voxel into multiple sub-voxels, treating the sub-voxels as new voxels, and returning to execute the step of determining the feature information corresponding to the voxel based on the feature points of the reprojected current frame target point cloud data in the voxel and the feature points of the historical feature point cloud map in the voxel, until the number of voxel divisions reaches a preset threshold.

[0010] In conjunction with the first aspect, in some implementations of the first aspect, the step of fusing multiple first point cloud sets to obtain a second point cloud set includes: for each of the multiple first point cloud sets, performing rasterization processing on the first point cloud set to obtain multiple point cloud grids corresponding to the first point cloud set; determining the target point cloud grid containing the target to be measured from the multiple point cloud grids; and obtaining the point cloud corresponding to the target point cloud grid from the multiple first point cloud sets to obtain the second point cloud set.

[0011] In conjunction with the first aspect, in some implementations of the first aspect, determining the target point cloud grid containing the target to be measured from the plurality of point cloud grids includes: determining the observation probability corresponding to each point cloud grid in the plurality of point cloud grids; performing probability fusion processing on the observation probabilities corresponding to point cloud grids at the same position in the plurality of first point cloud sets to obtain fusion probabilities corresponding to the plurality of point cloud grids; and deleting point cloud grids with fusion probabilities less than a preset probability threshold from the plurality of point cloud grids to obtain the target point cloud grid.

[0012] In conjunction with the first aspect, in some implementations of the first aspect, determining the observation probability corresponding to each of the plurality of point cloud grids includes: performing the following judgment on each of the plurality of point cloud grids: if the point cloud grid contains at least one of the at least one target to be measured, then the observation probability corresponding to the point cloud grid is determined to be a first target probability; if the point cloud grid does not contain any target to be measured, then the observation probability corresponding to the point cloud grid is determined to be a second target probability.

[0013] In conjunction with the first aspect, in some implementations of the first aspect, the first target probability includes a preset fixed probability or the confidence level of the target to be tested contained in the point cloud grid.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, determining the target point cloud set corresponding to the at least one target to be tested based on the second point cloud set includes: obtaining a first point cloud corresponding to a first target to be tested from the second point cloud set, wherein the first target to be tested includes any one of the at least one target to be tested; performing plane fitting on the first point cloud to obtain a first plane; determining the distance between each point in the first point cloud and the first plane; determining points in the first point cloud whose distance to the first plane is within a preset distance range as candidate points to obtain a candidate point set; and determining the target point cloud corresponding to the first target to be tested based on the candidate point set, wherein the target point cloud set includes the target point cloud corresponding to the first target to be tested.

[0015] In conjunction with the first aspect, in some implementations of the first aspect, after determining the target point cloud corresponding to the first target to be tested based on the candidate point set, the method further includes: performing polygon fitting processing on the target point cloud to obtain a two-dimensional projection shape corresponding to the first target to be tested.

[0016] In conjunction with the first aspect, in some implementations of the first aspect, the step of performing polygon fitting processing on the target point cloud to obtain a two-dimensional projection shape corresponding to the first target to be tested includes: determining the projection size of the target point cloud; if the projection size of the target point cloud is less than or equal to a preset size threshold, then performing convex hull fitting processing on the target point cloud to obtain a two-dimensional projection shape corresponding to the first target to be tested; if the projection size of the target point cloud is greater than the preset size threshold, then performing concave hull fitting processing on the target point cloud to obtain a two-dimensional projection shape corresponding to the first target to be tested.

[0017] In conjunction with the first aspect, in some implementations of the first aspect, obtaining the first point cloud set corresponding to the at least one target to be tested from the point cloud map based on the image detection result includes: obtaining the pose information corresponding to the target image data acquired by the target image acquisition device, and the calibration parameters corresponding to the target image acquisition device; projecting the point cloud map from the world coordinate system to the image coordinate system corresponding to the target image data based on the pose information and the calibration parameters to obtain the point cloud map in the image coordinate system; determining the pixel point cloud correspondence between the target image data and the point cloud map in the image coordinate system; and obtaining the first point cloud set corresponding to the at least one target to be tested from the point cloud map according to the pixel point cloud correspondence based on the image detection result.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: acquiring video data containing the at least one target to be tested; and acquiring multiple frames of images from the video data as multiple target image data.

[0019] In conjunction with the first aspect, in some implementations of the first aspect, the step of obtaining multiple frames of images from the video data as multiple target image data includes: obtaining consecutive multiple frames of images from the video data as multiple target image data at preset time intervals.

[0020] Secondly, one embodiment of this application provides a target detection device, comprising: an image detection module, configured to perform target detection on target image data to obtain an image detection result corresponding to at least one target to be tested; a point cloud acquisition module, configured to acquire a first point cloud set corresponding to the at least one target to be tested from a point cloud map constructed from multiple frames of target point cloud data corresponding to the target image data based on the image detection result; a fusion processing module, configured to perform fusion processing on multiple first point cloud sets to obtain a second point cloud set, wherein the multiple first point cloud sets include first point cloud sets acquired based on multiple target image data respectively; and a point cloud determination module, configured to determine a target point cloud set corresponding to the at least one target to be tested based on the second point cloud set.

[0021] Thirdly, one embodiment of this application provides a computer-readable storage medium storing a computer program for performing the target detection method described in the first aspect.

[0022] Fourthly, one embodiment of this application provides an electronic device, the electronic device comprising: a processor; a memory for storing processor-executable instructions; the processor being configured to perform the target detection method described in the first aspect.

[0023] Fifthly, one embodiment of this application provides a computer program product including instructions that, when executed on an electronic device, cause the electronic device to implement the target detection method described in the first aspect.

[0024] In this application, based on at least one target detected from target image data, a first point cloud set corresponding to the target is obtained from a point cloud map constructed from multiple frames of target point cloud data. Then, a more accurate second point cloud set is obtained by fusing multiple first point cloud sets. From this second point cloud set, a target point cloud set with high recall and high precision can be obtained. Thus, by leveraging the high recall of image detection, the target is initially located, and the high-resolution point cloud map accumulated from multiple frames of target point cloud data provides accurate spatial location information. Simultaneously, by fusing multiple point cloud sets corresponding to the at least one target, the detection precision of the at least one target can be improved probabilistically or spatially, thereby achieving high recall and high precision detection of small targets. Attached Figure Description

[0025] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0026] Figure 1 The diagram shown is a flowchart of a target detection method provided in an embodiment of this application.

[0027] Figure 2 The diagram shown is a schematic flowchart of an image point cloud association process provided in an embodiment of this application.

[0028] Figure 3 The diagram shown is a flowchart illustrating the first point cloud aggregation fusion process provided in an embodiment of this application.

[0029] Figure 4 The diagram shown is a schematic diagram of a point cloud grid provided in an embodiment of this application.

[0030] Figure 5 The diagram shown is a schematic representation of the occlusion effect provided in an embodiment of this application.

[0031] Figure 6 The diagram shown is a flowchart illustrating the process of determining a target point cloud set according to an embodiment of this application.

[0032] Figure 7 The diagram shown is a schematic diagram of the target detection device provided in an embodiment of this application.

[0033] Figure 8 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0034] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0035] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented even without certain specific details. In some instances, methods and means well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0036] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0037] Furthermore, the terms “first,” “second,” “third,” and “fourth” are used only for distinguishing descriptions and should not be interpreted as indicating or implying relative importance.

[0038] In the field of autonomous driving environmental perception technology, especially in road safety and quality inspection in scenarios with poor road conditions such as mining areas, accurately and reliably identifying small targets such as gravel, ruts, and potholes on the road surface is a key requirement for achieving safe navigation and operation. This requires the perception system not only to be able to detect small targets with a high recall rate, but also to provide high-precision three-dimensional position information corresponding to the target to support subsequent path planning and decision-making.

[0039] In relevant target detection schemes, target detection is generally based on sensor data of a single mode.

[0040] One approach involves first acquiring image data using a vision sensor, performing 2D object detection based on this data, and then converting the 2D object detection results into a Bird's Eye View (BEV). This method is sensitive to texture and shape information, thus achieving a relatively high object recall rate. However, due to its inherent lack of direct depth measurement capabilities, the accuracy of converting the 2D object detection results into real-world 3D location information is limited, especially when the target is far away or small. The conversion process introduces significant positioning errors, making it difficult to meet the requirements of high-precision 3D positioning. For example, for pothole detection on roads: when the camera is close to the pothole, it may occupy 50×50 pixels in the image, but at 30 meters away, it may shrink to 5x5 pixels. Direct vision detection can still barely identify it, but after conversion to BEV, a pothole 30 meters away from the camera may not even be as large as a voxel grid.

[0041] Another approach is to directly utilize the 3D point cloud data acquired by LiDAR for target detection. LiDAR can directly provide accurate distance information, which is beneficial for 3D positioning. However, due to the time gaps in its scanning, point cloud data is prone to missing small targets, resulting in a low recall rate for small target detection.

[0042] It is evident that existing solutions relying on a single sensor mode, due to the inherent limitations of their respective sensing principles, struggle to simultaneously achieve high levels in both target recall and detection accuracy, the two core performance indicators, when dealing with small target detection tasks on the road surface.

[0043] To address the aforementioned issues, this application provides a target detection method and electronic device. This method involves obtaining a first point cloud set corresponding to the target from a point cloud map constructed from multiple frames of target point cloud data, based on at least one target detected from target image data. Then, it fuses multiple first point cloud sets to obtain a more accurate second point cloud set. From this second point cloud set, a target point cloud set with high recall and high precision can be obtained. Thus, by leveraging the high recall of image detection, the target is initially located, and the high-resolution point cloud map accumulated from multiple frames of target point cloud data provides precise spatial location information. Simultaneously, by fusing multiple point cloud sets corresponding to the at least one target, the detection accuracy of the at least one target can be improved probabilistically or spatially, thereby achieving high recall and high precision detection of small targets.

[0044] The following is combined with Figures 1 to 6 The target detection method provided in this application is described in detail.

[0045] Figure 1 The diagram shown is a schematic flowchart of a target detection method provided in an embodiment of this application. This method can be applied to electronic devices, such as computers, vehicle-mounted systems, and other devices with data processing capabilities. Figure 1 As shown, the method may include the following steps.

[0046] S110, Target detection is performed on the target image data to obtain the image detection result corresponding to at least one target to be detected.

[0047] In some examples, the target image data can be data acquired by any type of image acquisition device that can be used for visual target detection. This target image data can be used to perform two-dimensional target detection to identify the location or region of at least one target contained within the image. For example, the target image data may include, but is not limited to: RGB images, grayscale images, infrared images acquired in real time by image acquisition devices such as vehicle-mounted cameras or surveillance cameras, or images extracted frame by frame from a continuous video stream. Furthermore, the target to be detected can be road targets contained in the target image data that need to be detected. These can be small targets such as stones, ruts, and potholes, or large targets such as vehicles, roadblocks, and pedestrians.

[0048] For example, target image data can be acquired by a target image acquisition device installed on an autonomous vehicle, and the target image data can be input into a target detection model. The target detection model can then identify and detect at least one target contained in the target image data, and output an image detection result. The image detection result may contain relevant information about the at least one target.

[0049] In one specific implementation, the image detection result may include the bounding box of each target in the image coordinate system, the pixel value (u, v) of the pixel where each target is located, and the semantic label and confidence score corresponding to each target. For example, for a stone detected from target image data, the output image detection result may include: the position coordinates of the polygon bounding box, the pixel value of the pixel in the area where the stone is located, the semantic label "stone", and a confidence score of 0.92. This image detection result can provide clear pixel-level information about the target for subsequent steps, thereby providing accurate guidance for finding the corresponding point cloud set from the point cloud map.

[0050] It should be noted that step S110 above provides a preliminary target candidate with high recall for the entire scheme of this application. The output image detection result is the starting point and guiding signal for all subsequent three-dimensional point cloud data association and fusion.

[0051] S120, based on the image detection results, obtain the first point cloud set corresponding to at least one target from the point cloud map constructed from the multi-frame target point cloud data corresponding to the target image data.

[0052] In some examples, target point cloud data can be a set of original 3D points collected by radar (such as lidar) and spatially and temporally correlated with target image data. It's important to note that at the data correlation level, there is a spatiotemporal synchronization relationship between target point cloud data and target image data. That is, at the same moment or within a very short time interval when acquiring a frame of target image data, the environment can be scanned multiple times by the onboard radar of the same perception platform (such as an autonomous vehicle), resulting in multiple corresponding frames of target point cloud data. This synchronization can be ensured through a unified timestamp system or trigger signal, thereby guaranteeing that the point cloud and the image depict the instantaneous state of the same physical scene.

[0053] Furthermore, a point cloud map can be a model representing the three-dimensional structure of an environment, constructed by fusing multiple frames of target point cloud data in a unified coordinate system. Specific methods for constructing a point cloud map from multiple frames of target point cloud data include, but are not limited to: directly accumulating multiple frames of target point cloud data in a unified coordinate system to form a point cloud map; or eliminating displacement errors between multiple frames of target point cloud data using a preset method, and then accumulating the error-eliminating multiple frames of target point cloud data in a unified coordinate system to obtain a point cloud map.

[0054] In some examples, the first point cloud set may be a set of three-dimensional points obtained from a point cloud map and initially associated with at least one target indicated in the image detection results. The methods for obtaining the first point cloud set include, but are not limited to: projecting the point cloud map onto the image coordinate system and selecting the corresponding three-dimensional point set in reverse based on the position of the target in the image detection results. This first point cloud set may contain points belonging to the actual target, or it may contain noise points that do not belong to the target due to projection errors, occlusion, or other reasons.

[0055] For example, multiple frames of target point cloud data corresponding to the target image data can be accumulated to obtain a point cloud map. The point cloud map is then projected onto the image coordinate system, and the corresponding three-dimensional point set in the point cloud map is selected in reverse based on the position of the target to be detected in the image detection results to obtain the first point cloud set.

[0056] S130, perform fusion processing on multiple first point cloud sets to obtain a second point cloud set, wherein the multiple first point cloud sets include first point cloud sets obtained based on multiple target image data respectively.

[0057] In this step, since the points contained in the first point cloud set do not accurately represent the target under test, the confidence of the real target point cloud can be enhanced probabilistically by complementing and verifying multiple frames of images. At the same time, noise points introduced by false detections in single frames, projection association errors, or instantaneous occlusions are suppressed, thereby significantly improving the accuracy (reducing false alarms) and recall (avoiding missed detections) of the three-dimensional detection of the target under test. Finally, a more reliable and complete three-dimensional point set, namely the second point cloud set, is output.

[0058] In some examples, for multiple target image data collected continuously, multiple first point cloud sets can be obtained through the aforementioned steps S110 and S120, and then the multiple first point cloud sets are fused to obtain a second point cloud set.

[0059] To further optimize the adaptability to dynamic scenes and the efficiency of data processing in the above embodiments, this application also provides the following preferred solutions. As mentioned above, the target image data can originate from a continuous video stream. Based on this, in some embodiments, the target detection method provided in this embodiment may further include: acquiring video data containing at least one target to be detected; and acquiring multiple frames of images from the video data as multiple target image data.

[0060] For example, the video data can be a continuous video stream acquired by an image acquisition device, such as video stream data containing an obstacle acquired by an onboard camera while a vehicle is in motion. For example, multiple frames containing at least one target can be obtained from the continuously acquired video data as multiple target image data. For instance, each frame can be obtained from 10 seconds of video data containing an obstacle as target image data. In this way, different angles of the obstacle can be shown in the different target image data, thereby obtaining more data about the obstacle and helping to more accurately detect and identify it.

[0061] Furthermore, in order to enable the system to sample and update the environment at a fixed and controllable frequency, avoid the huge computational load caused by processing each frame of video, and at the same time ensure the continuity of information acquisition, in some embodiments, the above-mentioned acquisition of multiple frames of images from video data as multiple target image data includes: acquiring consecutive multiple frames of images from video data as multiple target image data at preset time intervals.

[0062] In some examples, the preset time interval can be the image frame acquisition interval set in advance according to actual needs. It can be fixed or adaptive. For example, the time interval for acquiring image frames is shortened when the vehicle is traveling at high speed, and the time interval for acquiring image frames is lengthened when the vehicle is traveling at low speed or stationary.

[0063] For example, given 10 seconds of video data containing an obstacle, one frame can be extracted from the 10-second video stream every second, resulting in 10 frames. Each frame serves as a target image. This allows the system to operate stably on vehicle-mounted systems with limited computing resources, maintaining good real-time perception and balancing the trade-off between perception update frequency and computational overhead. Furthermore, the extracted frames can be positioned to display the obstacle from different angles, thus acquiring image data from more perspectives for better obstacle identification.

[0064] In addition, those skilled in the art will understand that the acquired image frames may not be strictly equally spaced, but rather selected are those with higher image clarity or those whose vehicle movement exceeds a certain distance from the previous processed frame, in order to obtain more valuable image data.

[0065] Furthermore, after obtaining multiple target image data sets, the first point cloud sets corresponding to each of these multiple target image data sets, each containing at least one target to be tested, can be fused to obtain a second point cloud set with higher recall and higher accuracy. The fusion processing methods include, but are not limited to: for the points contained in the multiple first point cloud sets, based on the probability or confidence that each point can truly represent the target to be tested, deleting points with lower probability or confidence and retaining points with higher probability or confidence.

[0066] S140, determine the target point cloud set corresponding to at least one target to be tested based on the second point cloud set.

[0067] In some examples, the second point cloud set can be a 3D point set with higher recall and higher precision obtained by fusing multiple first point cloud sets.

[0068] In one implementation, the second point cloud set can be directly used as the final target point cloud set for the at least one target to be tested. In another implementation, the second point cloud set can be further processed to extract the target to be tested within it at the instance level, thereby obtaining a cleaner and / or structured final 3D point set, i.e., the target point cloud set. Further processing methods include, but are not limited to: filtering out noise points such as ground points from the second point cloud set, or further fitting the point cloud clusters representing the target to be tested in the second point cloud set to obtain point cloud clusters with regular shapes. Thus, by further refining and / or structuring the second point cloud set, a clean and usable 3D representation of each target instance can be obtained, thereby achieving high-precision target detection.

[0069] The target detection method provided in this embodiment obtains a first point cloud set corresponding to the target from a point cloud map constructed from multiple frames of target point cloud data, based on at least one target detected from target image data. Then, it fuses multiple first point cloud sets to obtain a more accurate second point cloud set. From this second point cloud set, a target point cloud set with high recall and high precision can be obtained. Thus, by leveraging the high recall of image detection, the target is initially located, and the high-resolution point cloud map accumulated from multiple frames of target point cloud data provides accurate spatial location information. Simultaneously, by fusing multiple point cloud sets corresponding to the at least one target, the detection precision of the at least one target can be improved probabilistically or spatially, thereby achieving high recall and high precision detection of small targets.

[0070] The following is combined with Figure 2 This section details how the image detection results are associated with the first point cloud set in the point cloud map.

[0071] Figure 2 The diagram shown is a schematic flowchart of an image point cloud association process provided in an embodiment of this application. Figure 1 Extending from the illustrated embodiment Figure 2 The illustrated embodiment will be described in detail below. Figure 2 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0072] like Figure 2 As shown, the above-mentioned method of obtaining the first point cloud set corresponding to at least one target from the point cloud map constructed from the multi-frame target point cloud data corresponding to the target image data based on the image detection results (step S120) may specifically include the following steps: S210: For each frame of target point cloud data in the multi-frame target point cloud data corresponding to the target image data, the target point cloud data is projected from the radar coordinate system to the world coordinate system based on the initial transformation parameters to obtain multi-frame target point cloud data in the world coordinate system.

[0073] In some examples, the initial transformation parameters can be those used to transform point cloud data from its original radar coordinate system to a unified world coordinate system. These initial transformation parameters can be estimated based on radar calibration parameters and real-time pose information provided by the Inertial Navigation System (INS). For example, based on radar calibration parameters and real-time kinematic (RTK) data, the transformation matrices from the radar coordinate system to the Inertial Measurement Unit (IMU) coordinate system and from the IMU coordinate system to the world coordinate system can be determined, and these two transformation matrices can be used as the initial transformation parameters.

[0074] For example, for each frame of target point cloud data, when the radar acquires the target point cloud data, the RTK data is acquired according to the corresponding data acquisition timestamp, and then the transformation matrix from the IMU coordinate system to the world coordinate system is determined. At the same time, the transformation matrix from the radar coordinate system to the IMU coordinate system is determined using the radar calibration parameters. Then, the projection calculation is performed according to the following formula (1) to project the target point cloud data from the radar coordinate system to the world coordinate system, and the target point cloud data in the world coordinate system is obtained.

[0075] in, For points in the target point cloud data in the radar coordinate system, For points in the target point cloud data in the world coordinate system, This is the transformation matrix from the radar coordinate system to the IMU coordinate system. This is the transformation matrix from the IMU coordinate system to the world coordinate system.

[0076] S220 performs fusion processing on multiple frames of target point cloud data in the world coordinate system to obtain a point cloud map.

[0077] For example, after converting multiple frames of target point cloud data to the same world coordinate system, the multiple frames of target point cloud data that have been converted to the same world coordinate system can be fused and integrated into a unified, consistent and high-precision 3D environmental model, that is, a point cloud map.

[0078] Specifically, fusion processing can be implemented in various ways depending on different requirements for accuracy and computational resources. In one basic implementation, point clouds from different frames can be directly superimposed onto the same world coordinate system according to the chronological order of data acquisition or based on initial transformation parameters to form a preliminary point cloud map. This method is computationally efficient, but the accumulated errors in sensor pose estimation may cause "ghosting" or structural blurring in the point cloud map, which is particularly unfavorable for the accurate representation of small targets.

[0079] To overcome the aforementioned deficiencies and obtain a more accurate point cloud map, this application provides an optimized fusion processing method. In some embodiments, multi-frame target point cloud data includes current frame target point cloud data and historical frame target point cloud data. Based on this, step S220 may specifically include: acquiring a historical point cloud map and a historical feature point cloud map constructed based on historical frame target point cloud data in the world coordinate system; extracting features from the current frame target point cloud data to obtain a feature point cloud corresponding to the current frame target point cloud data; performing adjustment processing on the feature point cloud corresponding to the current frame target point cloud data based on the historical feature point cloud map to optimize the initial transformation parameters and obtain target transformation parameters; reprojecting the current frame target point cloud data to the world coordinate system based on the target transformation parameters to obtain reprojected current frame target point cloud data; and fusing the reprojected current frame target point cloud data with the historical point cloud map to obtain a point cloud map.

[0080] For example, the system can maintain a historical point cloud map and a historical feature point cloud map extracted from it. When new target point cloud data (i.e., the target point cloud data of the current frame) arrives, feature extraction can be performed on the target point cloud data of the current frame first, such as extracting its geometric features (e.g., surface features, line features, etc.) to obtain the feature point cloud of the current frame. Subsequently, the feature point cloud of the current frame is matched with the historical feature point cloud map to construct an optimization problem and solve for a more accurate transformation parameter, i.e., the target transformation parameter, to minimize the feature alignment error between the feature point cloud of the current frame and the historical feature point cloud map. This adjustment process effectively corrects the pose drift caused by relying solely on external positioning sources (e.g., RTK data). Finally, using the optimized target transformation parameter and according to the aforementioned formula (1), the target point cloud data of the current frame is re-transformed from the radar coordinate system to the world coordinate system and then integrated into the historical point cloud map, thereby achieving incremental and high-precision updates to the point cloud map.

[0081] In addition, the above process of feature extraction of the target point cloud data in the current frame may include, for example, the point set corresponding to the target point cloud data in the current frame. Its center of mass is The covariance matrix is ​​generated according to the following formula (2); after matrix decomposition of the covariance matrix, the eigenvalues ​​can be obtained: Among them, the line features satisfy The corresponding feature vector is Surface features satisfy , When the value is close to 0, the corresponding feature vector is .

[0082] in, Let covariance matrix be the variance matrix. The number of points contained in the target point cloud data. The point corresponding to the centroid. For the target point cloud data, the first One point.

[0083] Furthermore, the aforementioned adjustment process may specifically include: for the feature point cloud corresponding to the target point cloud data in the current frame, if it contains n feature points, then for any feature point... You can first search for the corresponding feature points in the historical feature point cloud map. Subsequently, the corresponding rotation matrix R and translation vector t are solved using the following optimization formula (3), and then the optimized target transformation parameters are re-determined based on the optimized R and t. For example, the transformation matrix from the IMU coordinate system to the world coordinate system can be re-determined based on the optimized R and t, and then combined with the transformation matrix from the radar coordinate system to the IMU coordinate system, the optimized target transformation parameters including these two transformation matrices can be obtained.

[0084] Where R is the rotation matrix and t is the translation vector. For the j-th feature point in the feature point cloud of the current frame, In historical feature point cloud maps and Corresponding feature points, Feature points in historical feature point cloud map The feature vector (such as the normal vector corresponding to line features or surface features). As can be seen from the above formula (3), the parameter optimization process is achieved by solving a function that minimizes the distance error. This function takes into account the distance from the current frame feature point to the corresponding feature point in the map and the degree of alignment with the feature vector.

[0085] In addition, the current frame target point cloud data, reprojected using target transformation parameters, is fused with the historical point cloud map. This process specifically includes: saving the reprojected current frame target point cloud data as a new data unit to a point cloud data queue used to maintain the point cloud map. This queue stores target point cloud data from the most recent preset number of frames in chronological order. The system can maintain the size of the point cloud map by controlling the maximum number of frames in this queue. Specifically, the system can use adaptive logic to manage the scale and real-time performance of the point cloud map: the external detection task can set the maximum number of frames in the map; when the number of data frames in the queue exceeds a preset frame threshold, the system automatically performs a deletion operation, deleting the oldest frame of target point cloud data.

[0086] It should be noted that if the historical point cloud map has not yet been established or the system has been initialized, the adjustment process can be omitted. The initial transformation parameters can be used to directly transform the target point cloud data of the current frame from the radar coordinate system to the world coordinate system. Then, the target point cloud data of the current frame in the world coordinate system can be added directly to the historical point cloud map as new data.

[0087] To make the feature matching and adjustment optimization process more accurate, thereby supporting long-term, robust pose optimization and suppressing cumulative drift, the historical feature point cloud map can also be updated synchronously. In some embodiments, the target detection method may further include: fusing the feature point cloud corresponding to the reprojected target point cloud data of the current frame with the historical feature point cloud map to obtain a feature point cloud map.

[0088] For example, the feature point cloud corresponding to the target point cloud data of the current frame after reprojection can be fused with the historical feature point cloud map by direct accumulation or optimized archiving, thereby obtaining an updated feature point cloud map, which can be used as the historical feature point cloud map of the target point cloud data of the next frame.

[0089] To improve the accuracy of feature fusion, a more refined strategy based on voxels and recursive partitioning can be used to fuse feature point clouds. Specifically, in some embodiments, the above-mentioned fusion processing of the feature point cloud corresponding to the reprojected current frame target point cloud data with the historical feature point cloud map to obtain a feature point cloud map may specifically include: dividing the feature point cloud space corresponding to the historical feature point cloud map into voxels to obtain multiple voxels; performing the following processing on each of the multiple voxels: determining the feature information corresponding to the voxel based on the feature points of the feature point cloud corresponding to the reprojected current frame target point cloud data in the voxel and the feature points of the historical feature point cloud map in the voxel; if the feature information corresponding to the voxel is the target feature information, then fusing the feature points in the voxel; if the feature information corresponding to the voxel is not the target feature information, then dividing the voxel into multiple sub-voxels, taking the sub-voxels as a new voxel, and returning to execute the determination of the feature information corresponding to the voxel based on the feature points of the feature point cloud corresponding to the reprojected current frame target point cloud data in the voxel and the feature points of the historical feature point cloud map in the voxel, until the number of voxel divisions reaches a preset threshold.

[0090] In some examples, the feature point cloud space corresponding to the historical feature point cloud map can also be the three-dimensional space in which the feature point cloud is located. A voxel can be a cubic spatial unit. In addition, target feature information can be obvious geometric features such as line features and surface features.

[0091] For example, the three-dimensional space where the feature point cloud is located is divided into voxels to obtain multiple voxels. For each voxel, the following operations are performed: the existing historical feature points (i.e., the feature points of the historical feature point cloud map in the voxel) and the newly added current frame feature points (i.e., the feature points of the feature point cloud corresponding to the current frame target point cloud data after reprojection in the voxel) are gathered together, and feature extraction is performed on the feature points contained in the voxel. The feature extraction process may include: calculating the centroid of the feature points contained in the voxel, determining the covariance matrix according to the aforementioned formula (2), performing eigenvalue decomposition on the covariance matrix, and obtaining eigenvalues. The corresponding feature vectors are then used. The geometric characteristics of the feature points within the voxel can be determined based on the distribution of feature values; for details, refer to the aforementioned extraction process for line and surface features. If the feature information corresponding to the voxel is the target feature information, such as a line or surface feature, all feature points within the voxel can be fused, for example, by archiving newly added current frame feature points within the voxel. If the feature information corresponding to the voxel is not the target feature information, such as not a line or surface feature, the voxel can be further divided into multiple sub-voxels, for example, into 8 sub-voxels. Then, the above operation is recursively repeated for each sub-voxel to find stable geometric features at a finer scale. Additionally, a division upper limit can be set for the recursive division, i.e., a preset number threshold, for example, a maximum of n divisions, to avoid infinite subdivision. When the number of voxel divisions reaches the preset number threshold, the recursion will stop and newly added, unfused feature points within the voxel will be discarded, even if the features within the sub-voxel are still insignificant.

[0092] In addition, to control the size of the feature point cloud map and maintain its relevance to the current scene, the system can perform deletion operations. For example, if the distance between a feature point region on the map and the current vehicle trajectory exceeds a preset threshold, the system will automatically delete the feature points in that region. This sliding window management method based on spatial distance ensures the real-time performance and computational efficiency of the feature point cloud map data.

[0093] In addition, in some implementations, when performing adjustment optimization, it is necessary to select a specific feature point in the feature point cloud of the current frame. Quickly locate corresponding feature points in historical feature point cloud maps. This operation can also be achieved through voxelized queries, and the specific process may include: based on... Given the position coordinates in the world coordinate system, directly calculate the coordinates of the voxel mesh to which it belongs, and then extract the archived feature points (such as the center point) from that voxel. ) and its eigenvectors (such as normal vectors) This information is used as the query result. Thus, thanks to the spatial index structure established during the addition operation, the matching and searching process for feature points becomes more efficient and accurate.

[0094] S230, Based on the image detection results, obtain the first point cloud set corresponding to at least one target to be tested from the point cloud map.

[0095] For example, the correspondence between the image and the point cloud can be used to back-project the target detected in the image detection result into the three-dimensional space, and extract all three-dimensional points falling within this back-projection space area from the high-precision point cloud map, thereby obtaining the set of three-dimensional points corresponding to each target detected in the image, that is, the first point cloud set.

[0096] To establish a precise correlation between two-dimensional image detection results and three-dimensional point cloud data, in some embodiments, step S230 may specifically include: acquiring the pose information corresponding to the target image data acquired by the target image acquisition device, and the calibration parameters corresponding to the target image acquisition device; based on the pose information and calibration parameters, projecting the point cloud map from the world coordinate system to the image coordinate system corresponding to the target image data to obtain the point cloud map in the image coordinate system; determining the pixel-point cloud correspondence between the target image data and the point cloud map in the image coordinate system; and based on the image detection results, acquiring a first point cloud set corresponding to at least one target to be tested from the point cloud map according to the pixel-point cloud correspondence.

[0097] In some examples, the target image acquisition device may be, for example, a vehicle-mounted camera for acquiring target image data. Pose information may include, for example, RTK data, and calibration parameters may be, for example, the camera matrix or intrinsic parameter matrix of the vehicle-mounted camera.

[0098] For example, when the vehicle-mounted camera acquires target image data, it can also simultaneously acquire the pose information of the vehicle-mounted camera, such as RTK data. Based on this pose information, the transformation matrix from the world coordinate system to the camera coordinate system can be determined. In addition, the calibration parameters of the vehicle-mounted camera, such as the camera matrix or intrinsic parameter matrix, can also be acquired. These calibration parameters define the transformation matrix from the camera coordinate system to the image coordinate system. Based on the transformation matrix from the world coordinate system to the camera coordinate system, each point in the point cloud map can be projected from the world coordinate system to the camera coordinate system to obtain a three-dimensional point P in the camera coordinate system. Then, according to the following formula (4), the three-dimensional point P in the camera coordinate system is projected to the two-dimensional image coordinate system to obtain a point cloud map in the image coordinate system.

[0099] in, For the point cloud map, the three-dimensional points are located in the camera coordinate system. This is either the camera matrix or the intrinsic parameter matrix. , For camera focal length, , Main point location, These are the pixel coordinates of a 3D point in a point cloud map in the image coordinate system.

[0100] Based on this, after obtaining the pixel coordinates of each point in the point cloud map in the image coordinate system, the pixel-point cloud correspondence between each pixel in the target image data and each point in the point cloud map can be determined. Then, using this correspondence, based on the pixel position of the target object contained in the image detection results, the points in the point cloud map associated with each target object can be determined, thus obtaining a first point cloud set corresponding to at least one target object. It should be noted that since the resolution of point clouds is usually higher than that of image files, and due to the camera projection principle, each pixel may be associated with multiple points. Since each target object in the image detection results is labeled with corresponding information, such as identification and semantic information, the points associated with each target object in the first point cloud set can also be labeled with the corresponding information.

[0101] In this embodiment, by uniformly registering and fusing multi-frame target point cloud data to the world coordinate system, a point cloud map with a unified spatial reference and significantly improved accuracy is constructed. This map effectively suppresses the spatial inconsistencies (such as ghosting and cumulative errors) caused by the sparsity of single-frame point cloud data and the simple accumulation of multi-frame point cloud data. It provides a high-resolution, highly consistent spatial basis for the accurate association between subsequent image detection results and 3D spatial information, thereby providing the ability to perform stable and high-precision 3D detection of small targets. In addition, by associating image detection results with point cloud data, the system can detect tiny targets on the image that are difficult to directly calculate in 3D and provide them with radar-level positioning accuracy, thereby significantly improving the accuracy of 3D detection of small targets.

[0102] The following is combined with Figure 3 This section details how to fuse multiple first point cloud sets to obtain a second point cloud set.

[0103] Figure 3 The diagram shown is a schematic flowchart of a first point cloud aggregation fusion processing procedure provided in an embodiment of this application. Figure 1 Extending from the illustrated embodiment Figure 3 The illustrated embodiment will be described in detail below. Figure 3 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0104] like Figure 3 As shown, the above-described fusion process of multiple first point cloud sets to obtain a second point cloud set (step S130) may specifically include the following steps: S310, for each of the multiple first point cloud sets, perform rasterization processing on the first point cloud set to obtain multiple point cloud rasteres corresponding to the first point cloud set.

[0105] For example, each first point cloud set can be subjected to the same rasterization process to obtain multiple point cloud grids, and each point cloud grid can be associated with multiple point clouds, for example, in... Figure 4 Of the nine point cloud grids shown, the point cloud associated with point cloud grid 41 is point cloud 42.

[0106] S320 identifies the target point cloud grid containing the target to be measured from multiple point cloud grids.

[0107] For example, point cloud grids that do not contain the target to be measured can be removed from the multiple point cloud grids, and redundant point cloud data associated with these point cloud grids can be deleted to obtain a relatively pure target point cloud grid that is only associated with point cloud data related to the target to be measured.

[0108] In some embodiments, step S320 may specifically include: determining the observation probability corresponding to each point cloud grid in the plurality of point cloud grids; performing probability fusion processing on the observation probabilities corresponding to the point cloud grids at the same position in the plurality of first point cloud sets to obtain the fusion probability corresponding to the plurality of point cloud grids; deleting point cloud grids with fusion probabilities less than a preset probability threshold from the plurality of point cloud grids to obtain the target point cloud grid.

[0109] In some examples, the observation probability corresponding to a point cloud grid can be the probability of observing the target in that point cloud grid. This observation probability can be determined based on the semantic information of the annotation corresponding to each point in the first point cloud set. For example, if a point is marked with target semantic information (such as "ruts", "stones", "potholes", etc.), then a higher probability can be set as the observation probability of the point cloud grid to which that point belongs based on the semantic information of that point.

[0110] For example, the technical essence of the above process is to quantify, statistically analyze, and make decisions on the three-dimensional spatial evidence (i.e., the first point cloud set) of the same target observed at different times. By aggregating information in a probabilistic form on discrete grid cells, it distinguishes between stable real target signals and occasional noise signals, thereby filtering out the real target signals. Specifically, after rasterizing each first point cloud set, each point cloud grid is assigned an observation probability representing the possibility of observing the target at that location. Then, for point cloud grids at the same spatial location, their corresponding observation probabilities in multiple first point cloud sets are fused according to a temporal fusion rule (e.g., current fusion probability = a). The fusion probability at the previous time step + (1-a) The current observation probability (where 'a' is the memory coefficient) is iteratively fused to obtain a fusion probability that accumulates over time. This probability value comprehensively reflects the confidence that the location has been consistently identified as a target in historical observations. Finally, a probability threshold (i.e., a preset probability threshold) is set. Grids with a fusion probability lower than this threshold are judged as having low confidence (possibly noise or accidental correlation) and are deleted; while grids with a fusion probability higher than the threshold are retained and identified as target point cloud grids.

[0111] In addition, in some embodiments, the determination of the observation probability corresponding to each point cloud grid in the plurality of point cloud grids may specifically include: performing the following judgment on each point cloud grid in the plurality of point cloud grids: if the point cloud grid contains at least one of the targets to be measured, then the observation probability corresponding to the point cloud grid is determined as the first target probability; if the point cloud grid does not contain any target to be measured, then the observation probability corresponding to the point cloud grid is determined as the second target probability.

[0112] For example, after rasterizing the current first point cloud set, the system traverses each point cloud raster and performs a judgment: if the raster contains at least one 3D point belonging to any target to be measured, it is determined that the current observation "discovered a target" at that location, and its observation probability is determined as the first target probability. Conversely, if the point cloud raster does not contain any target point cloud, it is determined that the current observation "did not discover a target" at that location, and its observation probability is determined as the second target probability.

[0113] In some embodiments, the first target probability may include a preset fixed probability or the confidence level of the target contained in the point cloud grid. The preset fixed probability may be a preset fixed probability value, such as 0.9, representing the basic confidence level of observing a target at this location. The confidence level of the target contained in the point cloud grid may be the confidence level output by the target detection model when detecting the target contained in the target image data, corresponding to each target. By associating and utilizing the confidence levels output by the image-based target detection model, the observation probability of the grid containing the associated point cloud of a target with a high confidence level in the image is correspondingly higher.

[0114] Alternatively, the probability of the second target can be a low, fixed probability value, such as 0.

[0115] S330: Obtain the point cloud corresponding to the target point cloud grid from multiple first point cloud sets to obtain the second point cloud set.

[0116] For example, after obtaining the target point cloud raster, the point cloud associated with the target point cloud raster can be restored to obtain a second point cloud set that contains only point cloud data related to the real target to be measured.

[0117] In this embodiment, multiple first point cloud sets are uniformly rasterized and discretized, and a high-probability target point cloud raster that persists across multiple frames is determined within the multiple point cloud rasteres. Then, the point cloud falling into the target point cloud raster is obtained as the second point cloud set, realizing raster-level temporal information fusion. This not only significantly suppresses false alarms caused by association errors or noise in single-frame observations, but also enhances the stable detection capability of real targets (especially small targets with sparse point clouds) through the accumulation of evidence from multiple frames. Thus, it synergistically solves the contradiction between accuracy and recall in small target detection.

[0118] Furthermore, due to the potential occlusion effect during projection from three dimensions to two dimensions, for example... Figure 5 As shown, the stone 51 on the ground is projected into the two-dimensional image 52. After mapping the image detection results of image 52 to the point cloud, during the fusion of multiple frames of point cloud data, the point cloud corresponding to the ground part behind the stone 51 that is occluded may also be counted as the point cloud corresponding to the stone 51 and accumulated, so that the point cloud corresponding to the stone 51 includes the ground part. Therefore, the second point cloud set can be further extracted or optimized at the instance level to obtain a more accurate target point cloud set.

[0119] The following is combined with Figure 6 This section details how to perform instance-level extraction or optimization on the second point cloud set to determine the target point cloud set.

[0120] Figure 6 The diagram shown is a flowchart illustrating the process of determining a target point cloud set according to an embodiment of this application. Figure 1 Extending from the illustrated embodiment Figure 6 The illustrated embodiment will be described in detail below. Figure 6 The illustrated embodiments and Figure 1 The differences between the embodiments shown are not repeated here, and the similarities are not repeated here.

[0121] like Figure 6 As shown, the above-mentioned determination of the target point cloud set corresponding to the at least one target to be tested based on the second point cloud set (step S140) may specifically include the following steps: S610, Obtain the first point cloud corresponding to the first target to be tested from the second point cloud set, wherein the first target to be tested includes any one of the at least one target to be tested.

[0122] For example, from the merged second set of point clouds, point cloud clusters belonging to a single independent target instance are separated by a clustering method (such as Euclidean clustering), and each point cloud cluster is defined as the first point cloud corresponding to a specific target (the first target).

[0123] S620, perform plane fitting on the first point cloud to obtain the first plane.

[0124] For example, a plane fitting is performed on the first point cloud to estimate a first plane that may be contained therein, which may be the ground occluded by the first target. Specifically, a robust fitting algorithm such as Random Sample Consensus (RANSAC) or other fitting algorithms may be used for plane fitting.

[0125] In a specific example, three non-collinear points are randomly selected from the first point cloud, and a planar model is constructed according to the following formula (5): in, Let be the normal vector of the plane, and D be a constant term. Calculate the distance from all points to the plane, and classify the points into inliers (points within the plane) and outliers (points not within the plane) based on a set distance threshold (e.g., 0.1 meters). Record the plane model with the most inliers. Repeat the above process until the maximum number of iterations is reached, or until a plane model that meets a preset condition is found, where the preset condition could be, for example, an inlier ratio exceeding a set threshold. Re-estimate the plane model using a target fitting algorithm based on all inliers to obtain the first plane, where the target fitting algorithm could be, for example, least squares fitting.

[0126] S630, determine the distance between each point in the first point cloud and the first plane.

[0127] For example, after determining the first plane, the distance between each point in the first point cloud and the first plane can be calculated. This distance can be a positive or negative distance; for example, if a point is above the first plane, the distance from that point to the first plane is positive, and if a point is below the first plane, the distance from that point to the first plane is negative.

[0128] S640, points in the first point cloud whose distance from the first plane is within a preset distance range are identified as candidate points, thus obtaining a set of candidate points.

[0129] In some examples, a minimum distance threshold (min_height) and a maximum distance threshold (max_height) can be preset, where the minimum distance threshold can be, for example, 0.1 meters and the maximum distance threshold can be, for example, 2.0 meters. Accordingly, the preset distance range can be, for example, [0.1, 2.1].

[0130] Exemplarily, for each point in the first point cloud, if the distance (d_signed) between the point and the first plane is within a preset distance range, for example, d_signed > min_height and d_signed < max_height, then the point can be determined as a candidate point. By analogy, a set of candidate points can be obtained.

[0131] S650, based on the set of candidate points, determine the target point cloud corresponding to the first target to be measured. The target point cloud set includes the target point cloud corresponding to the first target to be measured.

[0132] Exemplarily, a preset clustering algorithm can be used to cluster the sets of candidate points corresponding to different targets to be measured, clustering the candidate points into different clusters, and each cluster represents an independent target to be measured. Among them, the preset clustering algorithm can, for example, include the Density-Based Spatial Clustering of Applications with Noise (DBSCAN) algorithm or a distance-based clustering algorithm.

[0133] In addition, in order to convert the target point cloud into a structured, compact, and data that is convenient for the downstream system to directly use, in some embodiments, after determining the target point cloud corresponding to the first target to be measured based on the set of candidate points, the target detection method further includes: performing polygon fitting processing on the target point cloud to obtain a two-dimensional projection shape corresponding to the first target to be measured.

[0134] Exemplarily, the target point cloud corresponding to the first target to be measured that has undergone instance separation and plane filtering can be projected onto a horizontal plane (such as the ground) to obtain a two-dimensional point set, and then polygon fitting processing is performed on the two-dimensional point set to obtain a two-dimensional projection shape corresponding to the first target to be measured. This two-dimensional projection shape is a more concise, regular, and two-dimensional geometric expression that is easier for the downstream system to process after converting the target point cloud to BEV. Among them, the downstream system can, for example, include a platform visualization system, a vehicle-mounted system of an autonomous vehicle, etc.

[0135] In addition, in order to improve the fitting effect, in some embodiments, the above-mentioned performing polygon fitting processing on the target point cloud to obtain a two-dimensional projection shape corresponding to the first target to be measured includes: determining the projection size of the target point cloud; if the projection size of the target point cloud is less than or equal to a preset size threshold, then performing convex hull fitting processing on the target point cloud to obtain a two-dimensional projection shape corresponding to the first target to be measured; if the projection size of the target point cloud is greater than the preset size threshold, then performing concave hull fitting processing on the target point cloud to obtain a two-dimensional projection shape corresponding to the first target to be measured.

[0136] In some examples, the projection dimensions can be, for example, the length and width dimensions of the bounding box of the two-dimensional point set corresponding to the target point cloud.

[0137] For example, the projected size of the target point cloud (such as the length and width of the bounding box of a two-dimensional point set) is determined. Based on the comparison between the projected size and a preset size threshold, different polygon fitting algorithms are adaptively selected: if the projected size is less than or equal to the threshold (corresponding to small obstacles such as stones), a convex hull fitting algorithm is used to solve for the smallest convex polygon that can contain all projected points; if the projected size is greater than the threshold (corresponding to large or narrow obstacles such as tire tracks), a concave hull fitting algorithm (such as the Alpha-Shape algorithm) is used to capture possible concave contours and generate concave polygons that better fit the actual edges. Finally, the fitted polygon is the two-dimensional projected shape of the first target to be tested.

[0138] In this embodiment, by using plane fitting and distance threshold filtering, non-target point sets (such as occluded ground points) mixed in the target point cloud due to perspective projection correlation errors are effectively identified and separated, thereby significantly improving the geometric purity and accuracy of the three-dimensional spatial representation of the target point cloud, and further improving the accuracy of target detection.

[0139] The above text combined Figures 1 to 6 The embodiments of the target detection method of this application are described in detail below, in conjunction with... Figure 7 This application describes in detail embodiments of the target detection apparatus. It should be understood that the descriptions of the target detection method embodiments correspond to the descriptions of the target detection apparatus embodiments; therefore, any parts not described in detail can be found in the preceding method embodiments.

[0140] Figure 7 The diagram shown is a schematic representation of a target detection device according to an embodiment of this application. This device can be configured in an electronic device. Figure 7 As shown, the target detection device 700 provided in this application embodiment includes: The image detection module 710 is used to perform target detection on the target image data and obtain the image detection result corresponding to at least one target to be detected. The point cloud acquisition module 720 is used to acquire a first point cloud set corresponding to at least one target from the point cloud map constructed from multiple frames of target point cloud data corresponding to the target image data based on the image detection results. The fusion processing module 730 is used to perform fusion processing on multiple first point cloud sets to obtain a second point cloud set. The multiple first point cloud sets include first point cloud sets obtained based on multiple target image data respectively. The point cloud determination module 740 is used to determine a target point cloud set corresponding to at least one target under test based on a second point cloud set.

[0141] In one embodiment of this application, the point cloud acquisition module 720 is further configured to: for each frame of target point cloud data in the multi-frame target point cloud data corresponding to the target image data, project the target point cloud data from the radar coordinate system to the world coordinate system based on initial transformation parameters to obtain multi-frame target point cloud data in the world coordinate system; perform fusion processing on the multi-frame target point cloud data in the world coordinate system to obtain a point cloud map; and obtain a first point cloud set corresponding to at least one target to be tested from the point cloud map based on the image detection results.

[0142] In one embodiment of this application, the multi-frame target point cloud data includes the current frame target point cloud data and the historical frame target point cloud data. Accordingly, the point cloud acquisition module 720 is further configured to: acquire a historical point cloud map and a historical feature point cloud map constructed based on the historical frame target point cloud data in a world coordinate system; extract features from the current frame target point cloud data to obtain a feature point cloud corresponding to the current frame target point cloud data; perform adjustment processing on the feature point cloud corresponding to the current frame target point cloud data based on the historical feature point cloud map to optimize the initial transformation parameters and obtain target transformation parameters; reproject the current frame target point cloud data to the world coordinate system based on the target transformation parameters to obtain the reprojected current frame target point cloud data; and fuse the reprojected current frame target point cloud data with the historical point cloud map to obtain a point cloud map.

[0143] In one embodiment of this application, the point cloud acquisition module 720 is further configured to: fuse the feature point cloud corresponding to the reprojected target point cloud data of the current frame with the historical feature point cloud map to obtain a feature point cloud map.

[0144] In one embodiment of this application, the point cloud acquisition module 720 is further configured to: divide the feature point cloud space corresponding to the historical feature point cloud map into voxels to obtain multiple voxels; and perform the following processing on each of the multiple voxels: determine the feature information corresponding to the voxel based on the feature points of the feature point cloud corresponding to the current frame target point cloud data after reprojection in the voxel, and the feature points of the historical feature point cloud map in the voxel; if the feature information corresponding to the voxel is the target feature information, then fuse the feature points in the voxel; if the feature information corresponding to the voxel is not the target feature information, then divide the voxel into multiple sub-voxels, take the sub-voxels as a new voxel, and return to execute the determination of the feature information corresponding to the voxel based on the feature points of the feature point cloud corresponding to the current frame target point cloud data after reprojection in the voxel, and the feature points of the historical feature point cloud map in the voxel, until the number of voxel divisions reaches a preset threshold.

[0145] In one embodiment of this application, the fusion processing module 730 is further configured to: perform rasterization processing on each of the plurality of first point cloud sets to obtain a plurality of point cloud grids corresponding to the first point cloud set; determine the target point cloud grid containing the target to be measured from the plurality of point cloud grids; and obtain the point cloud corresponding to the target point cloud grid from the plurality of first point cloud sets to obtain a second point cloud set.

[0146] In one embodiment of this application, the fusion processing module 730 is further configured to: determine the observation probability corresponding to each point cloud grid in the plurality of point cloud grids; perform probability fusion processing on the observation probabilities corresponding to the point cloud grids at the same position in the plurality of first point cloud sets to obtain the fusion probability corresponding to the plurality of point cloud grids; and delete the point cloud grids whose fusion probability is less than a preset probability threshold from the plurality of point cloud grids to obtain the target point cloud grid.

[0147] In one embodiment of this application, the fusion processing module 730 is further configured to: perform the following judgment on each point cloud grid in the plurality of point cloud grids: if the point cloud grid contains at least one of the targets to be measured, then determine the observation probability corresponding to the point cloud grid as the first target probability; if the point cloud grid does not contain any target to be measured, then determine the observation probability corresponding to the point cloud grid as the second target probability.

[0148] In one embodiment of this application, the first target probability includes a preset fixed probability or the confidence level of the target to be tested contained in the point cloud grid.

[0149] In one embodiment of this application, the point cloud determination module 740 is further configured to: obtain a first point cloud corresponding to a first target to be tested from a second point cloud set, wherein the first target to be tested includes any one of at least one target to be tested; perform plane fitting on the first point cloud to obtain a first plane; determine the distance between each point in the first point cloud and the first plane; determine points in the first point cloud whose distance to the first plane is within a preset distance range as candidate points to obtain a candidate point set; and determine a target point cloud corresponding to the first target to be tested based on the candidate point set, wherein the target point cloud set includes the target point cloud corresponding to the first target to be tested.

[0150] In one embodiment of this application, the point cloud determination module 740 is further configured to: perform polygon fitting processing on the target point cloud to obtain a two-dimensional projection shape corresponding to the first target to be measured.

[0151] In one embodiment of this application, the point cloud determination module 740 is further configured to: determine the projection size of the target point cloud; if the projection size of the target point cloud is less than or equal to a preset size threshold, perform convex hull fitting processing on the target point cloud to obtain a two-dimensional projection shape corresponding to the first target to be measured; if the projection size of the target point cloud is greater than the preset size threshold, perform concave hull fitting processing on the target point cloud to obtain a two-dimensional projection shape corresponding to the first target to be measured.

[0152] In one embodiment of this application, the point cloud acquisition module 720 is further configured to: acquire the pose information corresponding to the target image data acquired by the target image acquisition device, and the calibration parameters corresponding to the target image acquisition device; based on the pose information and calibration parameters, project the point cloud map from the world coordinate system to the image coordinate system corresponding to the target image data to obtain the point cloud map in the image coordinate system; determine the pixel point cloud correspondence between the target image data and the point cloud map in the image coordinate system; and based on the image detection results, acquire the first point cloud set corresponding to at least one target to be tested from the point cloud map according to the pixel point cloud correspondence.

[0153] In one embodiment of this application, the target detection device 700 further includes: a video acquisition module for acquiring video data containing at least one target to be detected; and an image acquisition module for acquiring multiple frames of images from the video data as multiple target image data.

[0154] In one embodiment of this application, the image acquisition module is further configured to: acquire multiple consecutive frames of images from video data at preset time intervals as multiple target image data.

[0155] Below, for reference Figure 8 This describes an electronic device according to embodiments of the present application. Figure 8 The diagram shown is a structural schematic of an electronic device provided in an exemplary embodiment of this application.

[0156] like Figure 8 As shown, the electronic device 800 includes one or more processors 801 and memory 802.

[0157] The processor 801 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 800 to perform desired functions.

[0158] The memory 802 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 801 may execute the program instructions to implement the target detection methods of the various embodiments of this application described above and / or other desired functions. Various contents, such as image detection results, a first point cloud set, and a second point cloud set, may also be stored in the computer-readable storage medium.

[0159] In one example, the electronic device 800 may also include an input device 803 and an output device 804, which are interconnected via a bus system and / or other forms of connection mechanism (not shown).

[0160] The input device 803 may include, for example, a keyboard, a mouse, etc.

[0161] The output device 804 can output various information to the outside, including image detection results, a first point cloud set, a second point cloud set, etc. The output device 804 may include, for example, a display, a speaker, a printer, and a communication network and its connected remote output devices, etc.

[0162] Of course, for the sake of simplicity, Figure 8 Only some of the components of the electronic device 800 relevant to this application are shown in this illustration; components such as buses, input / output interfaces, etc., are omitted. In addition, the electronic device 800 may include any other suitable components depending on the specific application.

[0163] In addition to the methods and apparatus described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the target detection methods according to various embodiments of this application described above.

[0164] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0165] Furthermore, embodiments of this application may also be computer-readable storage media storing computer program instructions thereon, which, when executed by a processor, cause the processor to perform the steps in the target detection methods according to various embodiments of this application described above.

[0166] The computer-readable storage medium may be any combination of one or more readable media. A readable medium may be a readable signal medium or a readable storage medium. A readable storage medium may be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection having one or more wires, a portable disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof.

[0167] The basic principles of this application have been described above with reference to specific embodiments. However, it should be noted that the advantages, benefits, and effects mentioned in this application are merely examples and not limitations, and should not be considered as essential features of each embodiment of this application. Furthermore, the specific details disclosed above are for illustrative and facilitative purposes only, and are not limitations. These details do not limit the application to the necessity of employing the aforementioned specific details for implementation.

[0168] The block diagrams of devices, apparatuses, devices, and systems involved in this application are merely illustrative examples and are not intended to require or imply that they must be connected, arranged, or configured in the manner shown in the block diagrams. As those skilled in the art will recognize, these devices, apparatuses, devices, and systems can be connected, arranged, and configured in any manner. Words such as “comprising,” “including,” “having,” etc., are open-ended terms meaning “including but not limited to,” and are used interchangeably with them. The terms “or” and “and” as used herein refer to the terms “and / or,” and are used interchangeably with them unless the context clearly indicates otherwise. The term “such as” as used herein refers to the phrase “such as but not limited to,” and is used interchangeably with it.

[0169] It should also be noted that in the apparatus, equipment, and methods of this application, the components or steps can be disassembled and / or recombined. These disassemblies and / or recombinations should be considered as equivalent solutions of this application.

[0170] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use this application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the general principles defined herein can be applied to other aspects without departing from the scope of this application. Therefore, this application is not intended to be limited to the aspects shown herein, but rather to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0171] The above description has been given for purposes of illustration and description. Furthermore, this description is not intended to limit the embodiments of this application to the forms disclosed herein. Although numerous exemplary aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, alterations, additions, and sub-combinations thereof.

Claims

1. A target detection method, characterized in that, include: Target detection is performed on the target image data to obtain the image detection result corresponding to at least one target to be detected; Based on the image detection results, a first point cloud set corresponding to the at least one target to be tested is obtained from the point cloud map constructed from the multi-frame target point cloud data corresponding to the target image data. Multiple first point cloud sets are fused to obtain a second point cloud set, wherein the multiple first point cloud sets include the first point cloud sets obtained based on multiple target image data respectively; Based on the second point cloud set, a target point cloud set corresponding to the at least one target to be tested is determined.

2. The method according to claim 1, characterized in that, The step of obtaining a first point cloud set corresponding to the at least one target from a point cloud map constructed from multiple frames of target point cloud data corresponding to the target image data based on the image detection results includes: For each frame of target point cloud data in the multi-frame target point cloud data corresponding to the target image data, the target point cloud data is projected from the radar coordinate system to the world coordinate system based on the initial transformation parameters to obtain the multi-frame target point cloud data in the world coordinate system. The multi-frame target point cloud data are fused in the world coordinate system to obtain the point cloud map; Based on the image detection results, a first point cloud set corresponding to the at least one target to be tested is obtained from the point cloud map.

3. The method according to claim 2, characterized in that, The multi-frame target point cloud data includes current frame target point cloud data and historical frame target point cloud data; the fusion processing of the multi-frame target point cloud data in the world coordinate system to obtain the point cloud map includes: In the world coordinate system, acquire the historical point cloud map and the historical feature point cloud map constructed based on the historical frame target point cloud data; Feature extraction is performed on the target point cloud data of the current frame to obtain a feature point cloud corresponding to the target point cloud data of the current frame; Based on the historical feature point cloud map, the feature point cloud corresponding to the target point cloud data of the current frame is adjusted to optimize the initial transformation parameters and obtain the target transformation parameters. Based on the target transformation parameters, the target point cloud data of the current frame is reprojected to the world coordinate system to obtain the reprojected target point cloud data of the current frame; The point cloud data of the current frame after reprojection is fused with the historical point cloud map to obtain the point cloud map; The method further includes: The feature point cloud corresponding to the current frame target point cloud data after reprojection is fused with the historical feature point cloud map to obtain a feature point cloud map.

4. The method according to claim 3, characterized in that, The step of fusing the feature point cloud corresponding to the reprojected current frame target point cloud data with the historical feature point cloud map to obtain a feature point cloud map includes: The feature point cloud space corresponding to the historical feature point cloud map is divided into voxels to obtain multiple voxels. For each of the plurality of voxels, the following processing is performed: Based on the feature points of the feature point cloud corresponding to the current frame target point cloud data after reprojection in the voxel, and the feature points of the historical feature point cloud map in the voxel, the feature information corresponding to the voxel is determined. If the feature information corresponding to the voxel is the target feature information, then the feature points in the voxel are fused. If the feature information corresponding to the voxel is not the target feature information, the voxel is divided into multiple sub-voxels, and the sub-voxels are used as a new voxel. The process of determining the feature information corresponding to the voxel is then performed based on the feature points of the feature point cloud corresponding to the target point cloud data of the current frame after reprojection, and the feature points of the historical feature point cloud map in the voxel, until the number of voxel divisions reaches a preset threshold.

5. The method according to any one of claims 1 to 4, characterized in that, The process of fusing multiple first point cloud sets to obtain a second point cloud set includes: For each of the plurality of first point cloud sets, the first point cloud set is rasterized to obtain a plurality of point cloud raster corresponding to the first point cloud set; From the plurality of point cloud grids, determine the target point cloud grid in which the target to be measured exists; The second point cloud set is obtained by acquiring the point cloud corresponding to the target point cloud grid from the plurality of first point cloud sets; The step of determining the target point cloud grid containing the target to be measured from the plurality of point cloud grids includes: Determine the observation probability corresponding to each point cloud grid in the plurality of point cloud grids; The observation probabilities corresponding to the point cloud grids at the same position in the plurality of first point cloud sets are subjected to probability fusion processing to obtain the fusion probabilities corresponding to the plurality of point cloud grids respectively. Delete point cloud grids with a fusion probability less than a preset probability threshold from the plurality of point cloud grids to obtain the target point cloud grid.

6. The method according to claim 5, characterized in that, Determining the observation probability corresponding to each of the plurality of point cloud grids includes: For each of the plurality of point cloud grids, perform the following judgment: If the point cloud grid contains at least one of the at least one target to be measured, then the observation probability corresponding to the point cloud grid is determined as the first target probability. If the point cloud grid does not contain any target to be measured, then the observation probability corresponding to the point cloud grid is determined as the second target probability; The first target probability includes a preset fixed probability or the confidence level of the target to be tested contained in the point cloud grid.

7. The method according to any one of claims 1 to 6, characterized in that, The step of determining the target point cloud set corresponding to the at least one target to be tested based on the second point cloud set includes: Obtain a first point cloud corresponding to the first target to be tested from the second point cloud set, wherein the first target to be tested includes any one of the at least one target to be tested; Perform plane fitting on the first point cloud to obtain the first plane; Determine the distance between each point in the first point cloud and the first plane; Points in the first point cloud whose distance from the first plane is within a preset distance range are identified as candidate points, thus obtaining a candidate point set. Based on the candidate point set, a target point cloud corresponding to the first target to be tested is determined, wherein the target point cloud set includes the target point cloud corresponding to the first target to be tested.

8. The method according to claim 7, characterized in that, After determining the target point cloud corresponding to the first target to be tested based on the candidate point set, the method further includes: The target point cloud is subjected to polygon fitting processing to obtain a two-dimensional projection shape corresponding to the first target to be tested; The step of performing polygon fitting processing on the target point cloud to obtain a two-dimensional projection shape corresponding to the first target to be tested includes: Determine the projection size of the target point cloud; If the projection size of the target point cloud is less than or equal to a preset size threshold, then the target point cloud is subjected to convex hull fitting to obtain a two-dimensional projection shape corresponding to the first target to be tested. If the projected size of the target point cloud is greater than the preset size threshold, then the target point cloud is subjected to concave hull fitting to obtain a two-dimensional projected shape corresponding to the first target to be tested.

9. The method according to claim 2, characterized in that, The step of obtaining a first point cloud set corresponding to the at least one target to be tested from the point cloud map based on the image detection result includes: Acquire the pose information corresponding to the target image data acquired by the target image acquisition device, as well as the calibration parameters corresponding to the target image acquisition device; Based on the pose information and the calibration parameters, the point cloud map is projected from the world coordinate system to the image coordinate system corresponding to the target image data to obtain the point cloud map in the image coordinate system. In the image coordinate system, determine the pixel-point cloud correspondence between the target image data and the point cloud map; Based on the image detection results, a first point cloud set corresponding to the at least one target to be tested is obtained from the point cloud map according to the pixel point cloud correspondence.

10. The method according to any one of claims 1 to 9, characterized in that, Also includes: Acquire video data containing at least one target under test; Multiple frames of images are obtained from the video data as multiple target image data; The step of obtaining multiple frames of images from the video data as multiple target image data includes: Multiple consecutive frames of images are acquired from the video data at preset time intervals to form multiple target image data.

11. An electronic device, characterized in that, include: processor; Memory used to store the processor's executable instructions; The processor is used to execute the target detection method according to any one of claims 1 to 10.

Images