Target detection method and device, electronic equipment, storage medium and program product

Abstract

Embodiments of the present disclosure relate to a target detection method and device, electronic equipment, storage medium and program product. The method comprises: obtaining point cloud data of a laser radar and perception data of other perception devices; performing region proposal processing based on the point cloud data to obtain first region proposals; the first region proposals are used to represent candidate regions in which a target object detected by the laser radar is located; performing region proposal processing based on the perception data to obtain second region proposals; the second region proposals are used to represent candidate regions in which the target object detected by the other perception devices is located; and performing detection according to the first region proposals and the second region proposals to obtain a target detection result; the target detection result comprises a target region in which the target object is located and a category of the target object. The method can improve the perception ability of an autonomous vehicle.

Description

Technical Field

[0001] This disclosure relates to the field of target detection technology, and in particular to a target detection method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] With the development of automotive technology, autonomous driving technology has emerged. Currently, autonomous vehicles often use point cloud data collected by LiDAR (Light Detection and Ranging) to detect targets and thus perceive the surrounding environment.

[0003] Since environmental perception has a very important impact on the safe operation of autonomous vehicles, improving the perception capabilities of autonomous vehicles has become an urgent technical problem to be solved. Summary of the Invention

[0004] This disclosure provides a target detection method, apparatus, electronic device, storage medium, and program product that can be used to improve the perception capabilities of autonomous vehicles.

[0005] In a first aspect, embodiments of this disclosure provide a target detection method, the method comprising:

[0006] Acquire point cloud data from LiDAR and sensing data from other sensing devices;

[0007] Based on the point cloud data, a region proposal is performed to obtain a first region proposal; the first region proposal is used to represent the candidate region where the target object detected by the lidar is located.

[0008] Based on the perceived data, a region proposal is performed to obtain a second region proposal; wherein, the second region proposal is used to represent the candidate region where the target object is detected by the other sensing devices;

[0009] Detection is performed based on the first region proposal and the second region proposal to obtain target detection results; the target detection results include the target region where the target object is located and the category of the target object.

[0010] Secondly, embodiments of this disclosure provide a target detection device, the device comprising:

[0011] The data acquisition module is used to acquire point cloud data from LiDAR and sensing data from other sensing devices.

[0012] The first region proposal module is used to perform region proposal processing based on the point cloud data to obtain a first region proposal; the first region proposal is used to represent the candidate region where the target object detected by the lidar is located.

[0013] The second region proposal module is used to perform region proposal processing based on the sensing data to obtain a second region proposal; wherein, the second region proposal is used to represent the candidate region where the target object is detected by the other sensing devices;

[0014] The target detection module is used to perform detection based on the first region proposal and the second region proposal to obtain a target detection result; the target detection result includes the target region where the target object is located and the category of the target object.

[0015] Thirdly, embodiments of this disclosure provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0016] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0017] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0018] The target detection method, apparatus, electronic device, storage medium, and program product provided in this disclosure acquire point cloud data from LiDAR and perception data from other sensing devices; perform region proposal processing based on the point cloud data to obtain a first region proposal; perform region proposal processing based on the perception data to obtain a second region proposal; and perform detection based on the first and second region proposals to obtain a target detection result. This disclosure obtains all candidate regions where the target object is located based on the point cloud data from LiDAR and perception data from other sensing devices, and then performs target detection based on all candidate regions where the target object is located, thus obtaining a more accurate target detection result. Compared with the prior art that only uses LiDAR for perception, this disclosure combines other sensing devices with LiDAR for perception, which can improve the perception capability of autonomous vehicles. Attached Figure Description

[0019] Figure 1 This is a diagram illustrating the application environment of a target detection method in one embodiment;

[0020] Figure 2 This is a flowchart illustrating a target detection method in one embodiment;

[0021] Figure 3 This is a flowchart illustrating the detection steps based on a first region proposal and a second region proposal in one embodiment.

[0022] Figure 4 This is a flowchart illustrating the steps of obtaining a lidar feature map based on point cloud data in one embodiment.

[0023] Figure 5 This is a flowchart illustrating the training steps of a two-dimensional instance segmentation model in one embodiment.

[0024] Figure 6 This is a flowchart illustrating a target detection method in another embodiment;

[0025] Figure 7 This is one of the structural block diagrams of a target detection device in one embodiment;

[0026] Figure 8 This is a second structural block diagram of the target detection device in one embodiment;

[0027] Figure 9 This is the third structural block diagram of the target detection device in one embodiment;

[0028] Figure 10 This is a diagram of the internal structure of an electronic device in one embodiment. Detailed Implementation

[0029] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this disclosure and are not intended to limit the embodiments of this disclosure.

[0030] First, before introducing the technical solutions of the embodiments of this disclosure in detail, the technical background or evolution of the embodiments of this disclosure will be introduced. Typically, autonomous vehicles use point cloud data collected by LiDAR to detect targets and thus perceive the surrounding environment. Since environmental perception has a very important impact on the safe driving of autonomous vehicles, how to improve the perception capabilities of autonomous vehicles has become an urgent technical problem to be solved.

[0031] This disclosure provides a target detection scheme that acquires point cloud data from a LiDAR and perception data from other sensing devices; performs region proposal processing based on the point cloud data to obtain a first region proposal; performs region proposal processing based on the perception data to obtain a second region proposal; and performs detection based on the first and second region proposals to obtain a target detection result. This disclosure obtains all candidate regions where the target object is located based on the point cloud data from the LiDAR and perception data from other sensing devices, and then performs target detection based on all candidate regions where the target object is located. This results in a more accurate target detection result. Compared with the prior art that only uses LiDAR for perception, this disclosure combines other sensing devices with LiDAR for perception, which can improve the perception capability of autonomous vehicles. It should be noted that the applicant has devoted considerable creative effort to combining other sensing devices with LiDAR and the technical solutions described in the following embodiments.

[0032] The technical solutions involved in the embodiments of this disclosure will be described below in conjunction with the scenarios in which they are applied.

[0033] The target detection method provided in this disclosure can be applied to, for example, Figure 1 The application environment shown includes a vehicle 102, which is equipped with electronic devices, LiDAR, and other sensing devices. The LiDAR and other sensing devices can perceive the environment surrounding the vehicle, and the electronic devices control the vehicle based on the data collected by the LiDAR and other sensing devices, thus achieving autonomous driving. These other sensing devices may include, but are not limited to, millimeter-wave radar, various cameras, surround-view cameras, fisheye cameras, infrared cameras, and other image acquisition devices.

[0034] In one embodiment, such as Figure 2 As shown, a target detection method is provided, which is applied to... Figure 1 Taking an electronic device as an example, the explanation includes the following steps:

[0035] Step 201: Acquire point cloud data from the lidar and sensing data from other sensing devices.

[0036] During target detection, a lidar unit installed on the vehicle emits laser pulses into the surrounding environment and detects the returning laser pulses to obtain point cloud data. Other sensing devices installed on the vehicle perceive the environment around the vehicle and obtain sensing data. For example, a camera installed on the vehicle captures images of the surrounding environment to obtain environmental images. This disclosure does not limit the use of other sensing devices or sensing data.

[0037] Electronic devices acquire point cloud data from LiDAR and sensing data from other sensing devices.

[0038] Step 202: Perform region proposal processing based on point cloud data to obtain the first region proposal.

[0039] The first region proposal represents a candidate area where a target object detected by the lidar is located. The target object may include at least one of vehicles, people, animals, plants, and buildings. This disclosure does not limit the target object. If there are multiple target objects, the first region proposal may include the candidate areas where each target object is located, i.e., it includes multiple region detection boxes.

[0040] After acquiring point cloud data, the electronic device performs region proposal processing based on the point cloud data to obtain a first region proposal. Optionally, the point cloud data is input into a first region proposal network to obtain the first region proposal output by the first region proposal network.

[0041] Step 203: Perform region proposal processing based on the sensing data to obtain the second region proposal.

[0042] The second region proposal is used to represent the candidate regions where target objects detected by other sensing devices are located. If there are multiple target objects, the second region proposal includes the candidate regions where each target object is located, that is, it includes multiple region detection boxes.

[0043] After acquiring the sensing data, the electronic device performs region proposal processing based on the sensing data to obtain a second region proposal. Optionally, the sensing data is input into the second region proposal network to obtain the second region proposal output by the second region proposal network.

[0044] For example, the environmental image captured by the camera is input into the second region proposal network to obtain multiple region detection boxes output by the second region network.

[0045] The aforementioned region detection box may include the coordinates of the center point of the target object in the world coordinate system, the length, width, and height of the target object, and the rotation angle of the target object around the z-axis in the world coordinate system. This embodiment of the disclosure does not limit the region detection box and can be set according to actual conditions.

[0046] Step 204: Perform detection based on the first region proposal and the second region proposal to obtain the target detection result.

[0047] The target detection result includes the target area where the target object is located and the category of the target object. The target area where the target object is located can be represented by a region detection box, and the category of the target object can include categories such as vehicles, people, animals, plants, and buildings. Optionally, the category of the target object can be pixel-level, that is, each pixel has a corresponding category. This disclosure does not limit the category of the target object.

[0048] After obtaining the first and second region proposals, all candidate regions containing the target object are obtained. Detection is performed based on all candidate regions containing the target object to obtain the target region. Identification of the target region yields the category of the target object. This embodiment of the disclosure does not limit the detection method; it can be selected according to the actual situation.

[0049] In the above target detection method, point cloud data from a LiDAR and perception data from other sensing devices are acquired; region proposal processing is performed based on the point cloud data to obtain a first region proposal; region proposal processing is performed based on the perception data to obtain a second region proposal; and detection is performed based on the first and second region proposals to obtain a target detection result. This embodiment of the present disclosure obtains all candidate regions where the target object is located based on the point cloud data from the LiDAR and perception data from other sensing devices, and then performs target detection based on all candidate regions where the target object is located. This results in a more accurate target detection result. Compared with the prior art that only uses LiDAR for perception, this embodiment of the present disclosure combines other sensing devices with LiDAR for perception, which can improve the perception capability of autonomous vehicles.

[0050] In one embodiment, such as Figure 3 As shown, the process of obtaining target detection results by performing detection based on the first region proposal and the second region proposal may include the following steps:

[0051] Step 301: Merge the first region proposal and the second region proposal to obtain the target region proposal.

[0052] The target area proposal is used to represent the candidate area where the target object detected by lidar and other sensing devices is located.

[0053] Upon obtaining the first and second region proposals, two sets of region detection boxes for the target object are obtained. These two sets of region detection boxes can be merged to obtain the target region proposal. The target region proposal may include all region detection boxes from both sets, or it may include the union of the two sets of region detection boxes. This disclosure does not limit the merging process.

[0054] Understandably, merging the first and second region proposals involves combining the candidate regions where the target object detected by the lidar is located with the candidate regions where the target object is located detected by other sensing devices, thereby perceiving all candidate regions where the target object is located.

[0055] Step 302: Obtain the lidar feature map based on the point cloud data.

[0056] After acquiring point cloud data, the electronic device can perform feature extraction based on the point cloud data to obtain a lidar feature map. Feature extraction can employ a neural network model or other methods; this embodiment does not limit the specific method and can be configured according to actual conditions.

[0057] Step 303: Determine the object feature map corresponding to the target object based on the target area proposal and the lidar feature map.

[0058] After determining the target region proposal and the LiDAR feature map, the region of the target object in the LiDAR feature map can be determined based on the target region proposal. Then, the object feature map corresponding to the target object can be extracted from the LiDAR feature map based on this region.

[0059] Optionally, when extracting object feature maps from the LiDAR feature maps, a preset algorithm can be used to resample the feature maps of different target objects, sampling them to the same resolution so that subsequent instance segmentation can be performed in batches, thereby improving detection efficiency. The preset algorithm mentioned above may include a bilinear interpolation algorithm, and this disclosure does not limit the preset algorithm.

[0060] Step 304: Perform 3D instance segmentation on the object feature map to obtain the target detection result.

[0061] After determining the object feature map, 3D instance segmentation is performed on the object feature map. Optionally, the object feature map is input into a pre-trained 3D instance segmentation model to obtain the target detection result output by the 3D instance segmentation model.

[0062] Instance segmentation is a method that uses object detection algorithms to detect different instances and then uses semantic segmentation algorithms to label each instance region pixel by pixel.

[0063] Optionally, when calculations such as target tracking are required, the target detection results output by the 3D instance segmentation model can include only the target region where the target object is located, which can reduce the computational power requirements for subsequent target tracking calculations.

[0064] In the above embodiments, the first region proposal and the second region proposal are merged to obtain a target region proposal; a LiDAR feature map is obtained based on point cloud data; an object feature map corresponding to the target object is determined based on the target region proposal and the LiDAR feature map; and three-dimensional instance segmentation is performed on the object feature map to obtain the target detection result. This embodiment combines LiDAR with other sensing devices for environmental perception, which can improve the perception capabilities of autonomous vehicles.

[0065] In one embodiment, such as Figure 4As shown, the process of obtaining LiDAR feature maps from point cloud data can include the following steps:

[0066] Step 3021: Map the point cloud data according to the mapping relationship between the point cloud data and the depth map to obtain the lidar depth map.

[0067] The mapping relationship between point cloud data and depth map can be preset in electronic devices. After the point cloud data is acquired, the point cloud data is mapped according to the mapping relationship to obtain the corresponding lidar depth map.

[0068] The above mapping process may include: mapping each point in the point cloud data from the reference coordinate system to the spherical coordinate system to obtain the position of each point in the spherical coordinate system, and determining the angle between the position of the point in the spherical coordinate system and the x-axis of the spherical coordinate system as the first angle, and determining the angle between the position of the point in the spherical coordinate system and the z-axis of the spherical coordinate system as the second angle; using the first angle as the abscissa, the second angle as the ordinate, and the surface reflectivity information and / or depth information of the point in the point cloud as the pixel value to obtain the lidar depth map.

[0069] For example, if the reference coordinate system mentioned above is the world coordinate system, and the position of point 1 in the point cloud is (x1, y1, z1), then by performing mapping processing according to the mapping relationship, the position of point 1 in the spherical coordinate system is obtained as follows: Where r1 is the distance between the position of the point cloud point in the spherical coordinate system and the origin of the spherical coordinate system, and θ1 is the first included angle. Let θ1 be the second included angle. Using θ1 as the x-coordinate, Using θ1 as the ordinate and the surface reflectivity information of point 1 in the point cloud as the pixel value, a lidar depth map can be obtained. Alternatively, using θ1 as the abscissa, and... Using the surface reflectivity information of point 1 in the point cloud as the ordinate, and the depth information of point 1 in the point cloud as the pixel value of another channel, a lidar depth map can be obtained. The pixel values ​​can also use prior information such as projection error and classification results. This embodiment does not limit the pixel values ​​and can be set according to actual conditions.

[0070] For mechanical lidar, the mapping relationship between the point cloud data and the depth map is as follows: each row of pixels on the lidar depth map corresponds to a laser scan line (beam); each pixel in each row corresponds to a scan point of the lidar.

[0071] Optionally, before mapping the point cloud data, partial point cloud data collected by multiple LiDARs can be stitched together to obtain all the stitched point cloud data, and then the entire point cloud data can be mapped to obtain the LiDAR depth map.

[0072] Optionally, after obtaining the LiDAR depth map, the LiDAR depth map can be downsampled to reduce its resolution, thereby reducing the computational requirements for subsequent feature extraction processing.

[0073] Step 3022: Extract features from the LiDAR depth map to obtain the LiDAR feature map.

[0074] After obtaining the LiDAR depth map, feature extraction is performed on it. Optionally, the LiDAR depth map is input into a pre-trained backbone network for feature extraction, resulting in a LiDAR feature map output by the backbone network.

[0075] The backbone network described above can be a combination of convolutional neural networks and feature pyramid networks, or it can be a feature extraction network based on Transformer. This disclosure does not limit the backbone network.

[0076] In the above embodiments, the point cloud data is mapped according to the mapping relationship between point cloud data and depth map to obtain a LiDAR depth map; features are extracted from the LiDAR depth map to obtain a LiDAR feature map. This embodiment maps three-dimensional point cloud data into a two-dimensional LiDAR depth map. Compared with the currently commonly used method of meshing point cloud data, the LiDAR depth map of this embodiment not only retains more three-dimensional spatial information but is also more suitable for neural network models to learn from. Therefore, it can obtain a larger receptive field and more global information, thereby improving the detection accuracy of target detection.

[0077] In one embodiment, the process of performing region proposal processing based on point cloud data to obtain a first region proposal may include: inputting the LiDAR feature map into a pre-trained two-dimensional instance segmentation model to obtain the first region proposal output by the two-dimensional instance segmentation model.

[0078] Electronic devices can store pre-trained two-dimensional instance segmentation models. When performing region proposal processing based on point cloud data, the point cloud data is first mapped according to the mapping relationship between the point cloud data and the depth map to obtain a LiDAR depth map. Then, features are extracted from the LiDAR depth map to obtain a LiDAR feature map. Finally, the extracted LiDAR depth map is input into the two-dimensional instance segmentation model, which then outputs the first region proposal.

[0079] Understandably, by using a pre-trained two-dimensional instance segmentation model, the candidate area where the target object detected by the LiDAR is located can be obtained quickly and accurately.

[0080] Based on the above embodiments, such as Figure 5As shown, the training process of the above two-dimensional instance segmentation model can include the following steps:

[0081] Step 401: Obtain multiple sample point cloud data and the corresponding annotations for each sample point cloud data.

[0082] The label indicates the three-dimensional region where the sample object is located.

[0083] To acquire multiple sample point cloud data, manual annotation can be used to obtain the annotations corresponding to each sample point cloud data. Other annotation methods can also be used, and this disclosure does not limit the specific methods used.

[0084] Step 402: Obtain the corresponding sample feature map based on the point cloud data of each sample.

[0085] Based on the mapping relationship between point cloud data and depth maps, the point cloud data of each sample is mapped to obtain the corresponding sample depth map. Then, features are extracted from each sample depth map to obtain the corresponding sample feature map. The mapping processing and feature extraction can be referred to the description in the above embodiments, and will not be repeated here.

[0086] Step 403: Map the annotations according to the mapping relationship between point cloud data and depth map to obtain the two-dimensional region where the sample object is located.

[0087] Similarly, the annotations corresponding to the sample point cloud data are also mapped, that is, the 3D region where the sample object is located is mapped to a 2D space to obtain the 2D region where the sample object is located. The 2D region where the sample object is located can be represented by a region detection box that surrounds the sample object.

[0088] Optionally, the annotations corresponding to the sample point cloud data also include a three-dimensional segmentation mask. After mapping the annotations, a two-dimensional segmentation mask can also be obtained.

[0089] Understandably, given existing 3D annotations, 2D annotations can be obtained quickly and easily based on the mapping relationship, thereby improving model training efficiency.

[0090] Step 404: Train the model based on the sample feature map and the two-dimensional region where the sample object is located to obtain a two-dimensional instance segmentation model.

[0091] After determining the two-dimensional sample feature map and the two-dimensional region where the sample object is located, the sample feature map is used as the model input for model training. Then, the loss value between the model output and the two-dimensional region where the sample object is located is determined. If the loss value does not meet the preset convergence condition, the model parameters are adjusted, and training continues. Training ends when the loss value meets the preset convergence condition, and the model at the end of training is defined as the two-dimensional instance segmentation model. This embodiment does not limit the preset convergence condition.

[0092] In the above embodiments, multiple sample point cloud data and corresponding annotations for each sample point cloud data are acquired; corresponding sample feature maps are obtained based on each sample point cloud data; the annotations are mapped according to the mapping relationship between point cloud data and depth maps to obtain the two-dimensional region where the sample object is located; and a model is trained based on the sample feature maps and the two-dimensional region where the sample object is located to obtain a two-dimensional instance segmentation model. This embodiment of the present disclosure pre-trains a two-dimensional instance segmentation model. When performing region proposal processing based on point cloud data, the two-dimensional instance segmentation model can quickly and accurately obtain the first region proposal, thus improving the detection efficiency of object detection.

[0093] In one embodiment, the process of performing region proposal processing based on sensing data to obtain a second region proposal may include: performing target detection on the sensing data to obtain a third region proposal; and performing coordinate transformation processing on the third region proposal according to the positional relationship between other sensing devices and the lidar, and the mapping relationship between point cloud data and depth map, to obtain a second region proposal.

[0094] A pre-defined detection algorithm is used to detect targets in the perceived data, resulting in a third region proposal. Based on the intrinsic and extrinsic parameters of other sensing devices and the LiDAR, as well as the positional relationship between the other sensing devices and the LiDAR, a coordinate transformation relationship between the perceived data and the point cloud data can be established. This transformation relationship allows the third region proposal to be converted to the point cloud data coordinate system. Then, based on the mapping relationship between the point cloud data and the depth map, the region proposal in the point cloud data coordinate system is mapped to the depth map coordinate system, resulting in a second region proposal.

[0095] Taking environmental images as the perceived data as an example, target detection is performed on the environmental images to obtain a third region proposal. This third region proposal is then transformed into the point cloud coordinate system and then into the depth map coordinate system to obtain a second region proposal. Understandably, camera vision algorithms can provide very robust and long-range detection results; that is, when the perceived data is image data, the third region proposal has high accuracy.

[0096] In the above embodiments, target detection is performed on the perceived data to obtain a third region proposal; based on the positional relationship between other sensing devices and the lidar, and the mapping relationship between point cloud data and depth map, the third region proposal is subjected to coordinate transformation processing to obtain a second region proposal. This embodiment transforms the third region proposal detected from the perceived data into a second region proposal in the depth map coordinate system, facilitating the merging of the second and first region proposals to obtain all candidate regions where the target object is located, thus providing a detection basis for target detection.

[0097] In one embodiment, such as Figure 6 As shown, a target detection method is provided, which is applied to... Figure 1 Taking an electronic device as an example, the explanation includes the following steps:

[0098] Step 501: Acquire point cloud data from the lidar and sensing data from other sensing devices.

[0099] Step 502: Map the point cloud data according to the mapping relationship between the point cloud data and the depth map to obtain the lidar depth map.

[0100] Step 503: Extract features from the lidar depth map to obtain the lidar feature map.

[0101] Step 504: Input the LiDAR feature map into the pre-trained two-dimensional instance segmentation model to obtain the first region proposal output by the two-dimensional instance segmentation model.

[0102] Step 505: Target detection is performed on the sensing data to obtain a third region proposal; coordinate transformation is performed on the third region proposal based on the positional relationship between other sensing devices and the lidar, as well as the mapping relationship between point cloud data and depth map, to obtain a second region proposal.

[0103] Step 506: Merge the first region proposal and the second region proposal to obtain the target region proposal.

[0104] Step 507: Determine the object feature map corresponding to the target object based on the target area proposal and the lidar feature map.

[0105] Step 508: Perform 3D instance segmentation on the object feature map to obtain the target detection result.

[0106] In the above embodiments, point cloud data from a LiDAR and perception data from other sensing devices are acquired. The point cloud data is then mapped to a LiDAR depth map, and features are extracted from the LiDAR depth map to obtain a LiDAR feature map. Next, a first region proposal is obtained using two-dimensional instance segmentation and the LiDAR feature map. During the determination of the first region proposal, a second region proposal is determined based on the perception data. Subsequently, the first and second region proposals are merged to obtain a target region proposal; then, an object feature map of the target object is extracted from the LiDAR feature map; finally, detection is performed based on the target region proposal and the object feature map to obtain the target detection result. Because this embodiment combines other sensing devices with LiDAR for environmental perception, it can improve the perception capabilities of autonomous vehicles.

[0107] It should be understood that, although Figures 2 to 6The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figures 2 to 6 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.

[0108] In one embodiment, such as Figure 7 As shown, a target detection device is provided, comprising:

[0109] The data acquisition module 601 is used to acquire point cloud data from the lidar and sensing data from other sensing devices.

[0110] The first region proposal module 602 is used to perform region proposal processing based on point cloud data to obtain a first region proposal; the first region proposal is used to represent the candidate region where the target object detected by the lidar is located.

[0111] The second region proposal module 603 is used to perform region proposal processing based on sensing data to obtain a second region proposal; wherein, the second region proposal is used to represent the candidate region where the target object detected by other sensing devices is located;

[0112] The target detection module 604 is used to perform detection based on the first region proposal and the second region proposal to obtain the target detection result; the target detection result includes the target region where the target object is located and the category of the target object.

[0113] In one embodiment, such as Figure 8 As shown, the target detection module 604 includes:

[0114] The merging submodule 6041 is used to merge the first region proposal and the second region proposal to obtain the target region proposal; the target region proposal is used to represent the candidate region where the target object detected by the lidar and other sensing devices is located.

[0115] The first feature map acquisition submodule 6042 is used to acquire lidar feature maps based on point cloud data;

[0116] The second feature map acquisition submodule 6043 is used to determine the object feature map corresponding to the target object based on the target region proposal and the lidar feature map.

[0117] The detection submodule 6044 is used to perform 3D instance segmentation on the object feature map to obtain the target detection result.

[0118] In one embodiment, the first feature map acquisition submodule 6042 is specifically used to perform mapping processing on the point cloud data according to the mapping relationship between the point cloud data and the depth map to obtain the lidar depth map; and to perform feature extraction on the lidar depth map to obtain the lidar feature map.

[0119] In one embodiment, the first feature map acquisition submodule 6042 is specifically used to input the LiDAR depth map into a pre-trained backbone network for feature extraction, thereby obtaining the LiDAR feature map output by the backbone network.

[0120] In one embodiment, the first feature map acquisition submodule 6042 is specifically used to map each point cloud point in the point cloud data from the reference coordinate system to the spherical coordinate system, obtain the position of each point cloud point in the spherical coordinate system, and determine the angle between the position of the point cloud point in the spherical coordinate system and the x-axis of the spherical coordinate system as the first angle, and determine the angle between the position of the point cloud point in the spherical coordinate system and the z-axis of the spherical coordinate system as the second angle; using the first angle as the abscissa, the second angle as the ordinate, and the surface reflectivity information and / or depth information of the point cloud point as the pixel value, a lidar depth map is obtained.

[0121] In one embodiment, the detection submodule 6044 is specifically used to input the object feature map into a pre-trained 3D instance segmentation model to obtain the target detection result output by the 3D instance segmentation model.

[0122] In one embodiment, the first region proposal module 602 is specifically used to input the lidar feature map into a pre-trained two-dimensional instance segmentation model to obtain the first region proposal output by the two-dimensional instance segmentation model.

[0123] In one embodiment, such as Figure 9 As shown, it also includes:

[0124] The sample acquisition module 605 is used to acquire multiple sample point cloud data and the corresponding annotations for each sample point cloud data; the annotations are the three-dimensional regions where the sample objects are located.

[0125] The sample feature map acquisition module 606 is used to acquire the corresponding sample feature map based on the point cloud data of each sample.

[0126] The annotation mapping module 607 is used to map the annotations according to the mapping relationship between point cloud data and depth map to obtain the two-dimensional region where the sample object is located;

[0127] The model training module is used to train the model based on the sample feature map and the two-dimensional region where the sample object is located, so as to obtain a two-dimensional instance segmentation model.

[0128] In one embodiment, the second region proposal module 602 is specifically used to perform target detection on the sensing data to obtain a third region proposal; and to perform coordinate transformation processing on the third region proposal based on the positional relationship between other sensing devices and the lidar, as well as the mapping relationship between point cloud data and depth map, to obtain the second region proposal.

[0129] Specific limitations regarding the target detection device can be found in the limitations of the target detection method described above, and will not be repeated here. Each module in the aforementioned target detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the electronic device in hardware form, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0130] Figure 10 This is a block diagram illustrating an electronic device 1300 according to an exemplary embodiment. For example, the electronic device 1300 may be a vehicle central control unit, a mobile phone, a digital broadcasting terminal, a messaging device, a tablet device, a personal digital assistant, etc.

[0131] Reference Figure 10 The electronic device 1300 may include one or more of the following components: a processing component 1302, a memory 1304, a power supply component 1306, a multimedia component 1308, an audio component 1310, an input / output (I / O) interface 1312, a sensor component 1314, and a communication component 1316. The memory stores computer programs or instructions that run on the processor.

[0132] Processing component 1302 typically controls the overall operation of electronic device 1300, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 1302 may include one or more processors 1320 to execute instructions to perform all or part of the steps of the methods described above. Furthermore, processing component 1302 may include one or more modules to facilitate interaction between processing component 1302 and other components. For example, processing component 1302 may include a multimedia module to facilitate interaction between multimedia component 1308 and processing component 1302.

[0133] Memory 1304 is configured to store various types of data to support the operation of electronic device 1300. Examples of such data include instructions for any application or method operating on electronic device 1300, contact data, phonebook data, messages, pictures, videos, etc. Memory 1304 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0134] Power supply component 1306 provides power to various components of electronic device 1300. Power supply component 1306 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 1300.

[0135] Multimedia component 1308 includes a touch display screen that provides an output interface between the electronic device 1300 and the user. In some embodiments, the touch display screen may include a liquid crystal display (LCD) and a touch panel (TP). The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of a touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 1308 includes a front-facing camera and / or a rear-facing camera. When the electronic device 1300 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0136] Audio component 1310 is configured to output and / or input audio signals. For example, audio component 1310 includes a microphone (MIC) configured to receive external audio signals when electronic device 1300 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 1304 or transmitted via communication component 1316. In some embodiments, audio component 1310 also includes a speaker for outputting audio signals.

[0137] I / O interface 1312 provides an interface between processing component 1302 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0138] Sensor assembly 1314 includes one or more sensors for providing state assessments of various aspects of electronic device 1300. For example, sensor assembly 1314 may detect the on / off state of electronic device 1300, the relative positioning of components such as the display and keypad of electronic device 1300, changes in position of electronic device 1300 or a component of electronic device 1300, the presence or absence of user contact with electronic device 1300, the orientation or acceleration / deceleration of electronic device 1300, and temperature changes of electronic device 1300. Sensor assembly 1314 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 1314 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 1314 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0139] Communication component 1316 is configured to facilitate wired or wireless communication between electronic device 1300 and other devices. Electronic device 1300 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 1316 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 1316 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0140] In an exemplary embodiment, the electronic device 1300 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the target detection method described above.

[0141] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 1304 including instructions, which can be executed by a processor 1320 of an electronic device 1300 to perform the above-described method. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0142] In an exemplary embodiment, a computer program product is also provided, which, when executed by a processor, can implement the above-described methods. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, some or all of the above-described methods can be implemented, wholly or partially, according to the processes or functions described in the embodiments of this disclosure.

[0143] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0144] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0145] The above-described embodiments are merely illustrative of several implementation methods of the present disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present disclosure embodiments, and these all fall within the protection scope of the present disclosure embodiments. Therefore, the protection scope of the patent for the embodiments of the present disclosure should be determined by the appended claims.

Claims

1. A target detection method characterized by, The method comprises: acquiring point cloud data of a laser radar and perception data of other perception devices; performing region proposal processing based on the point cloud data to obtain first region proposals; the first region proposals are used to represent candidate regions in which target objects detected by the laser radar are located; performing region proposal processing based on the perception data to obtain second region proposals; the second region proposals are used to represent candidate regions in which the target objects detected by the other perception devices are located; performing detection according to the first region proposals and the second region proposals to obtain target detection results; the target detection results comprise target regions in which the target objects are located and categories of the target objects; wherein the performing detection according to the first region proposals and the second region proposals to obtain target detection results comprises: performing merging processing on the first region proposals and the second region proposals to obtain target region proposals; the target region proposals are used to represent candidate regions in which the target objects detected by the laser radar and the other perception devices are located; acquiring a laser radar feature map according to the point cloud data; determining a region of the target object in the laser radar feature map according to the target region proposals, and extracting an object feature map corresponding to the target object from the laser radar feature map according to the region, and resampling feature maps of different target objects in the extraction process; performing three-dimensional instance segmentation on the object feature map to obtain the target detection results.

2. The method of claim 1, wherein, The acquiring a laser radar feature map according to the point cloud data comprises: performing mapping processing on the point cloud data according to a mapping relationship between the point cloud data and a depth map to obtain a laser radar depth map; performing feature extraction on the laser radar depth map to obtain the laser radar feature map.

3. The method of claim 2, wherein, The performing feature extraction on the laser radar depth map to obtain the laser radar feature map comprises: inputting the laser radar depth map into a pre-trained backbone network to perform feature extraction, and obtaining the laser radar feature map output by the backbone network.

4. The method of claim 2, wherein, The performing mapping processing on the point cloud data according to a mapping relationship between the point cloud data and a depth map to obtain a laser radar depth map comprises: mapping each point cloud point in the point cloud data from a reference coordinate system to a spherical coordinate system to obtain a position of each point cloud point in the spherical coordinate system, and determining a first angle between the position of the point cloud point in the spherical coordinate system and an x-axis of the spherical coordinate system as a first angle, and determining a second angle between the position of the point cloud point in the spherical coordinate system and a z-axis of the spherical coordinate system as a second angle; taking the first angle as a horizontal coordinate, the second angle as a vertical coordinate, surface reflectivity information and / or depth information of the point cloud point as pixel values to obtain the laser radar depth map.

5. The method of claim 1, wherein, The performing three-dimensional instance segmentation on the object feature map to obtain the target detection results comprises: inputting the object feature map into a pre-trained three-dimensional instance segmentation model to obtain the target detection results output by the three-dimensional instance segmentation model.

6. The method according to any one of claims 1 to 5, characterized in that, The region proposal processing based on the point cloud data obtains a first region proposal, and the region proposal processing based on the perception data obtains a second region proposal. The laser radar feature map is input into a pre-trained two-dimensional instance segmentation model to obtain the first region proposal output by the two-dimensional instance segmentation model.

7. The method of claim 6, wherein, The training process of the two-dimensional instance segmentation model includes: Obtaining a plurality of sample point cloud data and labels corresponding to each sample point cloud data; the label is a three-dimensional region where a sample object is located; According to each sample point cloud data, a sample feature map is obtained; According to the mapping relationship between the point cloud data and the depth map, the label is mapped to obtain a two-dimensional region where the sample object is located; According to the sample feature map and the two-dimensional region where the sample object is located, model training is performed to obtain the two-dimensional instance segmentation model.

8. The method according to any one of claims 1 to 5, characterized in that, The region proposal processing based on the perception data obtains a second region proposal, and the region proposal processing based on the perception data obtains a second region proposal. The target detection is performed on the perception data to obtain a third region proposal; According to the positional relationship between the other perception device and the laser radar, and the mapping relationship between the point cloud data and the depth map, the third region proposal is subjected to coordinate conversion processing to obtain the second region proposal.

9. A target detection apparatus characterized by comprising: The device includes: A data acquisition module is configured to acquire point cloud data of a laser radar and perception data of other perception devices; A first region proposal module is configured to perform region proposal processing based on the point cloud data to obtain a first region proposal; the first region proposal is used to represent a candidate region where a target object detected by the laser radar is located; A second region proposal module is configured to perform region proposal processing based on the perception data to obtain a second region proposal; the second region proposal is used to represent a candidate region where the target object detected by the other perception device is located; A target detection module is configured to perform detection according to the first region proposal and the second region proposal to obtain a target detection result; the target detection result includes a target region where the target object is located and a category of the target object; The target detection module includes: A merging submodule is configured to perform merging processing on the first region proposal and the second region proposal to obtain a target region proposal; the target region proposal is used to represent a candidate region where the target object detected by the laser radar and the other perception device is located; A first feature map acquisition submodule is configured to acquire a laser radar feature map according to the point cloud data; A second feature map acquisition submodule is configured to determine a region of the target object in the laser radar feature map according to the target region proposal, and extract an object feature map corresponding to the target object from the laser radar feature map according to the region; and in the extraction process, the feature maps of different target objects are resampled; A detection submodule is configured to perform three-dimensional instance segmentation on the object feature map to obtain the target detection result.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method of any one of claims 1 to 8.

11. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method of any one of claims 1 to 8.

12. A computer program product comprising a computer program, characterized in that, The computer program, which when executed by the processor, implements the steps of the method of any one of claims 1-8.

Images